2024-04-25T06:53:04.644 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-25T06:53:04.644 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} --name nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi --owner scheduled_nmordech@teuthology --archive /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672922 --verbose -- /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672922/orig.config.yaml 2024-04-25T06:53:04.666 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T06:53:04.746 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672922 branch: main description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: null first_in_suite: false job_id: '7672922' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f workunit: branch: wip-nitzan-osd-recovery-standalone-test-wait-for-too-full sha1: a08304379f921e5e229e92c75ede8800059313ff owner: scheduled_nmordech@teuthology priority: 50 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 7777 sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f sleep_before_teardown: 0 suite: rados:standalone suite_branch: wip-nitzan-osd-recovery-standalone-test-wait-for-too-full suite_path: /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa suite_relpath: qa suite_repo: https://github.com/NitzanMordhai/ceph.git suite_sha1: a08304379f921e5e229e92c75ede8800059313ff targets: smithi134.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGRz+nAW6n/awnsGzulMSDhdR1nEJl6Q+sXCov4cVpwKHg6zrIzSQKQosJkSAFeuYTtPYR7ProL2pjfuUk5Wytw= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-25_06:28:28 tube: smithi user: nmordech verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-25T06:53:04.746 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa; will attempt to use it 2024-04-25T06:53:04.747 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_NitzanMordhai_ceph_a08304379f921e5e229e92c75ede8800059313ff/qa/tasks 2024-04-25T06:53:04.747 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-25T06:53:04.748 INFO:teuthology.task.internal:Checking packages... 2024-04-25T06:53:04.770 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f' 2024-04-25T06:53:04.770 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-25T06:53:04.770 INFO:teuthology.packaging:ref: None 2024-04-25T06:53:04.770 INFO:teuthology.packaging:tag: None 2024-04-25T06:53:04.770 INFO:teuthology.packaging:branch: main 2024-04-25T06:53:04.770 INFO:teuthology.packaging:sha1: c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T06:53:04.770 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-04-25T06:53:05.087 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3253-g4e213a58-1jammy 2024-04-25T06:53:05.089 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-25T06:53:05.100 INFO:teuthology.task.internal:no buildpackages task found 2024-04-25T06:53:05.100 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-25T06:53:05.129 INFO:teuthology.task.internal:Saving configuration 2024-04-25T06:53:05.138 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-25T06:53:05.152 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-25T06:53:05.174 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi134.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672922', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-25 06:47:25.670244', 'locked_by': 'scheduled_nmordech@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPYCI+XtNd36sI3+ZlwIMU4VvwC2kXuyGXLcD6WwODMR'} 2024-04-25T06:53:05.174 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-25T06:53:05.185 INFO:teuthology.task.internal:roles: ubuntu@smithi134.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-04-25T06:53:05.185 INFO:teuthology.run_tasks:Running task console_log... 2024-04-25T06:53:05.229 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fbdab9e83a0>, signals=[15]) 2024-04-25T06:53:05.230 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-25T06:53:05.240 INFO:teuthology.task.internal:Opening connections... 2024-04-25T06:53:05.240 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi134.front.sepia.ceph.com 2024-04-25T06:53:05.242 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:53:05.319 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-25T06:53:05.327 DEBUG:teuthology.orchestra.run.smithi134:> uname -m 2024-04-25T06:53:05.431 INFO:teuthology.orchestra.run.smithi134.stdout:x86_64 2024-04-25T06:53:05.431 DEBUG:teuthology.orchestra.run.smithi134:> cat /etc/os-release 2024-04-25T06:53:05.478 INFO:teuthology.orchestra.run.smithi134.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-04-25T06:53:05.478 INFO:teuthology.orchestra.run.smithi134.stdout:NAME="Ubuntu" 2024-04-25T06:53:05.478 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_ID="22.04" 2024-04-25T06:53:05.478 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-04-25T06:53:05.479 INFO:teuthology.orchestra.run.smithi134.stdout:VERSION_CODENAME=jammy 2024-04-25T06:53:05.479 INFO:teuthology.orchestra.run.smithi134.stdout:ID=ubuntu 2024-04-25T06:53:05.479 INFO:teuthology.orchestra.run.smithi134.stdout:ID_LIKE=debian 2024-04-25T06:53:05.479 INFO:teuthology.orchestra.run.smithi134.stdout:HOME_URL="https://www.ubuntu.com/" 2024-04-25T06:53:05.479 INFO:teuthology.orchestra.run.smithi134.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-04-25T06:53:05.479 INFO:teuthology.orchestra.run.smithi134.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-04-25T06:53:05.479 INFO:teuthology.orchestra.run.smithi134.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-04-25T06:53:05.479 INFO:teuthology.orchestra.run.smithi134.stdout:UBUNTU_CODENAME=jammy 2024-04-25T06:53:05.480 INFO:teuthology.lock.ops:Updating smithi134.front.sepia.ceph.com on lock server 2024-04-25T06:53:05.503 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-25T06:53:05.514 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-25T06:53:05.525 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-25T06:53:05.526 DEBUG:teuthology.orchestra.run.smithi134:> test '!' -e /home/ubuntu/cephtest 2024-04-25T06:53:05.530 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-25T06:53:05.537 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-25T06:53:05.538 DEBUG:teuthology.orchestra.run.smithi134:> test -z $(ls -A /var/lib/ceph) 2024-04-25T06:53:05.580 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-25T06:53:05.613 INFO:teuthology.run_tasks:Running task kernel... 2024-04-25T06:53:05.629 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-25T06:53:05.629 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-25T06:53:05.630 DEBUG:teuthology.orchestra.run.smithi134:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-25T06:53:05.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T06:53:05.634 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-25T06:53:05.682 INFO:teuthology.orchestra.run.smithi134.stdout:5.15.0-56-generic 2024-04-25T06:53:05.682 INFO:teuthology.task.kernel:Running kernel on smithi134: 5.15.0-56-generic 2024-04-25T06:53:05.683 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get clean 2024-04-25T06:53:05.805 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get update 2024-04-25T06:53:05.971 INFO:teuthology.orchestra.run.smithi134.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [110 kB] 2024-04-25T06:53:06.072 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T06:53:06.139 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [119 kB] 2024-04-25T06:53:06.326 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [452 kB] 2024-04-25T06:53:06.465 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,394 kB] 2024-04-25T06:53:06.500 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [109 kB] 2024-04-25T06:53:06.548 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [243 kB] 2024-04-25T06:53:06.554 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [11.4 kB] 2024-04-25T06:53:06.555 INFO:teuthology.orchestra.run.smithi134.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [1,773 kB] 2024-04-25T06:53:06.606 INFO:teuthology.orchestra.run.smithi134.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [36.5 kB] 2024-04-25T06:53:06.607 INFO:teuthology.orchestra.run.smithi134.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [300 kB] 2024-04-25T06:53:06.613 INFO:teuthology.orchestra.run.smithi134.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T06:53:06.641 INFO:teuthology.orchestra.run.smithi134.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [601 kB] 2024-04-25T06:53:06.641 INFO:teuthology.orchestra.run.smithi134.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [619 kB] 2024-04-25T06:53:06.647 INFO:teuthology.orchestra.run.smithi134.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [848 kB] 2024-04-25T06:53:06.688 INFO:teuthology.orchestra.run.smithi134.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [163 kB] 2024-04-25T06:53:06.689 INFO:teuthology.orchestra.run.smithi134.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [16.8 kB] 2024-04-25T06:53:06.690 INFO:teuthology.orchestra.run.smithi134.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-04-25T06:53:06.690 INFO:teuthology.orchestra.run.smithi134.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-04-25T06:53:06.690 INFO:teuthology.orchestra.run.smithi134.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-04-25T06:53:06.691 INFO:teuthology.orchestra.run.smithi134.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [260 B] 2024-04-25T06:53:06.841 INFO:teuthology.orchestra.run.smithi134.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,610 kB] 2024-04-25T06:53:06.994 INFO:teuthology.orchestra.run.smithi134.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [303 kB] 2024-04-25T06:53:07.009 INFO:teuthology.orchestra.run.smithi134.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [16.1 kB] 2024-04-25T06:53:07.010 INFO:teuthology.orchestra.run.smithi134.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [38.1 kB] 2024-04-25T06:53:07.011 INFO:teuthology.orchestra.run.smithi134.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [1,830 kB] 2024-04-25T06:53:07.072 INFO:teuthology.orchestra.run.smithi134.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [311 kB] 2024-04-25T06:53:07.081 INFO:teuthology.orchestra.run.smithi134.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [520 B] 2024-04-25T06:53:07.245 INFO:teuthology.orchestra.run.smithi134.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,070 kB] 2024-04-25T06:53:07.878 INFO:teuthology.orchestra.run.smithi134.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [700 kB] 2024-04-25T06:53:07.916 INFO:teuthology.orchestra.run.smithi134.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [244 kB] 2024-04-25T06:53:07.928 INFO:teuthology.orchestra.run.smithi134.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [22.1 kB] 2024-04-25T06:53:07.930 INFO:teuthology.orchestra.run.smithi134.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,472 B] 2024-04-25T06:53:07.930 INFO:teuthology.orchestra.run.smithi134.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [42.7 kB] 2024-04-25T06:53:07.932 INFO:teuthology.orchestra.run.smithi134.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.4 kB] 2024-04-25T06:53:07.933 INFO:teuthology.orchestra.run.smithi134.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [472 B] 2024-04-25T06:53:07.933 INFO:teuthology.orchestra.run.smithi134.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [90.8 kB] 2024-04-25T06:53:07.961 INFO:teuthology.orchestra.run.smithi134.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [82.9 kB] 2024-04-25T06:53:07.966 INFO:teuthology.orchestra.run.smithi134.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.0 kB] 2024-04-25T06:53:08.131 INFO:teuthology.orchestra.run.smithi134.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-04-25T06:53:08.214 INFO:teuthology.orchestra.run.smithi134.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [18.5 kB] 2024-04-25T06:53:08.298 INFO:teuthology.orchestra.run.smithi134.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [29.8 kB] 2024-04-25T06:53:08.380 INFO:teuthology.orchestra.run.smithi134.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.2 kB] 2024-04-25T06:53:08.412 INFO:teuthology.orchestra.run.smithi134.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [644 B] 2024-04-25T06:53:12.658 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 13.3 MB in 3s (5,246 kB/s) 2024-04-25T06:53:14.115 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-25T06:53:14.133 DEBUG:teuthology.orchestra.run.smithi134:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T06:53:14.194 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-25T06:53:14.394 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2024-04-25T06:53:14.394 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2024-04-25T06:53:14.569 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T06:53:14.569 INFO:teuthology.orchestra.run.smithi134.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T06:53:14.570 INFO:teuthology.orchestra.run.smithi134.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T06:53:14.570 INFO:teuthology.orchestra.run.smithi134.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T06:53:14.570 INFO:teuthology.orchestra.run.smithi134.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-04-25T06:53:14.570 INFO:teuthology.orchestra.run.smithi134.stdout: python2.7-minimal 2024-04-25T06:53:14.570 INFO:teuthology.orchestra.run.smithi134.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T06:53:14.571 INFO:teuthology.orchestra.run.smithi134.stdout:The following additional packages will be installed: 2024-04-25T06:53:14.572 INFO:teuthology.orchestra.run.smithi134.stdout: linux-generic linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T06:53:14.572 INFO:teuthology.orchestra.run.smithi134.stdout: linux-headers-generic linux-image-5.15.0-105-generic 2024-04-25T06:53:14.572 INFO:teuthology.orchestra.run.smithi134.stdout: linux-modules-5.15.0-105-generic linux-modules-extra-5.15.0-105-generic 2024-04-25T06:53:14.573 INFO:teuthology.orchestra.run.smithi134.stdout:Suggested packages: 2024-04-25T06:53:14.573 INFO:teuthology.orchestra.run.smithi134.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-04-25T06:53:14.573 INFO:teuthology.orchestra.run.smithi134.stdout:Recommended packages: 2024-04-25T06:53:14.573 INFO:teuthology.orchestra.run.smithi134.stdout: thermald 2024-04-25T06:53:14.619 INFO:teuthology.orchestra.run.smithi134.stdout:The following NEW packages will be installed: 2024-04-25T06:53:14.619 INFO:teuthology.orchestra.run.smithi134.stdout: linux-headers-5.15.0-105 linux-headers-5.15.0-105-generic 2024-04-25T06:53:14.620 INFO:teuthology.orchestra.run.smithi134.stdout: linux-image-5.15.0-105-generic linux-modules-5.15.0-105-generic 2024-04-25T06:53:14.620 INFO:teuthology.orchestra.run.smithi134.stdout: linux-modules-extra-5.15.0-105-generic 2024-04-25T06:53:14.621 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages will be upgraded: 2024-04-25T06:53:14.621 INFO:teuthology.orchestra.run.smithi134.stdout: linux-generic linux-headers-generic linux-image-generic 2024-04-25T06:53:14.801 INFO:teuthology.orchestra.run.smithi134.stdout:3 upgraded, 5 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T06:53:14.801 INFO:teuthology.orchestra.run.smithi134.stdout:Need to get 113 MB of archives. 2024-04-25T06:53:14.801 INFO:teuthology.orchestra.run.smithi134.stdout:After this operation, 583 MB of additional disk space will be used. 2024-04-25T06:53:14.801 INFO:teuthology.orchestra.run.smithi134.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-105-generic amd64 5.15.0-105.115 [22.7 MB] 2024-04-25T06:53:16.329 INFO:teuthology.orchestra.run.smithi134.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-105-generic amd64 5.15.0-105.115 [11.5 MB] 2024-04-25T06:53:16.639 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-105-generic amd64 5.15.0-105.115 [63.9 MB] 2024-04-25T06:53:18.816 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.105.102 [1,694 B] 2024-04-25T06:53:18.816 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.105.102 [2,530 B] 2024-04-25T06:53:18.816 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105 all 5.15.0-105.115 [12.3 MB] 2024-04-25T06:53:19.474 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-105-generic amd64 5.15.0-105.115 [2,857 kB] 2024-04-25T06:53:19.591 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.105.102 [2,382 B] 2024-04-25T06:53:19.991 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 113 MB in 5s (22.8 MB/s) 2024-04-25T06:53:20.169 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-modules-5.15.0-105-generic. 2024-04-25T06:53:22.138 INFO:teuthology.orchestra.run.smithi134.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-04-25T06:53:22.142 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../0-linux-modules-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:53:22.224 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:24.400 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-image-5.15.0-105-generic. 2024-04-25T06:53:24.421 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../1-linux-image-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:53:24.507 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:24.961 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-105-generic. 2024-04-25T06:53:24.981 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:53:25.010 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:32.024 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../3-linux-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:53:32.153 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:53:32.494 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:53:32.615 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-image-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:53:32.934 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-headers-5.15.0-105. 2024-04-25T06:53:32.956 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../5-linux-headers-5.15.0-105_5.15.0-105.115_all.deb ... 2024-04-25T06:53:33.000 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T06:53:38.941 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package linux-headers-5.15.0-105-generic. 2024-04-25T06:53:38.978 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../6-linux-headers-5.15.0-105-generic_5.15.0-105.115_amd64.deb ... 2024-04-25T06:53:39.015 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:42.521 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.105.102_amd64.deb ... 2024-04-25T06:53:42.646 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking linux-headers-generic (5.15.0.105.102) over (5.15.0.56.54) ... 2024-04-25T06:53:42.996 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-headers-5.15.0-105 (5.15.0-105.115) ... 2024-04-25T06:53:43.109 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-headers-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:43.235 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-headers-generic (5.15.0.105.102) ... 2024-04-25T06:53:43.363 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:44.707 INFO:teuthology.orchestra.run.smithi134.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:53:44.707 INFO:teuthology.orchestra.run.smithi134.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-04-25T06:53:44.707 INFO:teuthology.orchestra.run.smithi134.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:53:44.707 INFO:teuthology.orchestra.run.smithi134.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-105-generic 2024-04-25T06:53:44.874 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-modules-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:46.327 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-modules-extra-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:48.261 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-image-generic (5.15.0.105.102) ... 2024-04-25T06:53:48.404 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up linux-generic (5.15.0.105.102) ... 2024-04-25T06:53:48.529 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for linux-image-5.15.0-105-generic (5.15.0-105.115) ... 2024-04-25T06:53:48.588 INFO:teuthology.orchestra.run.smithi134.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-04-25T06:53:48.588 INFO:teuthology.orchestra.run.smithi134.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:54:02.556 INFO:teuthology.orchestra.run.smithi134.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-04-25T06:54:02.556 INFO:teuthology.orchestra.run.smithi134.stdout:Sourcing file `/etc/default/grub' 2024-04-25T06:54:02.582 INFO:teuthology.orchestra.run.smithi134.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T06:54:02.644 INFO:teuthology.orchestra.run.smithi134.stdout:Generating grub configuration file ... 2024-04-25T06:54:02.942 INFO:teuthology.orchestra.run.smithi134.stdout:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:54:02.962 INFO:teuthology.orchestra.run.smithi134.stdout:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:54:03.366 INFO:teuthology.orchestra.run.smithi134.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:54:03.372 INFO:teuthology.orchestra.run.smithi134.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T06:54:03.445 INFO:teuthology.orchestra.run.smithi134.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T06:54:03.451 INFO:teuthology.orchestra.run.smithi134.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T06:54:03.594 INFO:teuthology.orchestra.run.smithi134.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T06:54:03.595 INFO:teuthology.orchestra.run.smithi134.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T06:54:03.595 INFO:teuthology.orchestra.run.smithi134.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T06:54:03.635 INFO:teuthology.orchestra.run.smithi134.stdout:done 2024-04-25T06:54:04.124 DEBUG:teuthology.orchestra.run.smithi134:> dpkg -s linux-image-generic 2024-04-25T06:54:04.140 INFO:teuthology.orchestra.run.smithi134.stdout:Package: linux-image-generic 2024-04-25T06:54:04.140 INFO:teuthology.orchestra.run.smithi134.stdout:Status: install ok installed 2024-04-25T06:54:04.140 INFO:teuthology.orchestra.run.smithi134.stdout:Priority: optional 2024-04-25T06:54:04.140 INFO:teuthology.orchestra.run.smithi134.stdout:Section: kernel 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout:Installed-Size: 21 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: amd64 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout:Source: linux-meta 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout:Version: 5.15.0.105.102 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout:Recommends: thermald 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout:Description: Generic Linux kernel image 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout: This package will always depend on the latest generic kernel image 2024-04-25T06:54:04.141 INFO:teuthology.orchestra.run.smithi134.stdout: available. 2024-04-25T06:54:04.141 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-105-generic 2024-04-25T06:54:04.142 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-25T06:54:04.142 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-25T06:54:04.142 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-04-25T06:54:04.142 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi134.front.sepia.ceph.com, path=None, version=distro) 2024-04-25T06:54:04.142 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get clean 2024-04-25T06:54:04.348 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get update 2024-04-25T06:54:04.487 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T06:54:04.593 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T06:54:04.699 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T06:54:04.759 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T06:54:05.904 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-25T06:54:05.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-04-25T06:54:05.973 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-25T06:54:06.156 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2024-04-25T06:54:06.156 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2024-04-25T06:54:06.311 INFO:teuthology.orchestra.run.smithi134.stdout:linux-image-generic is already the newest version (5.15.0.105.102). 2024-04-25T06:54:06.311 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T06:54:06.311 INFO:teuthology.orchestra.run.smithi134.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T06:54:06.312 INFO:teuthology.orchestra.run.smithi134.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T06:54:06.312 INFO:teuthology.orchestra.run.smithi134.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T06:54:06.312 INFO:teuthology.orchestra.run.smithi134.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T06:54:06.312 INFO:teuthology.orchestra.run.smithi134.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T06:54:06.312 INFO:teuthology.orchestra.run.smithi134.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T06:54:06.351 INFO:teuthology.orchestra.run.smithi134.stdout:0 upgraded, 0 newly installed, 0 to remove and 315 not upgraded. 2024-04-25T06:54:06.353 DEBUG:teuthology.orchestra.run.smithi134:> dpkg -s linux-image-generic 2024-04-25T06:54:06.413 INFO:teuthology.orchestra.run.smithi134.stdout:Package: linux-image-generic 2024-04-25T06:54:06.413 INFO:teuthology.orchestra.run.smithi134.stdout:Status: install ok installed 2024-04-25T06:54:06.413 INFO:teuthology.orchestra.run.smithi134.stdout:Priority: optional 2024-04-25T06:54:06.413 INFO:teuthology.orchestra.run.smithi134.stdout:Section: kernel 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout:Installed-Size: 21 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout:Maintainer: Ubuntu Kernel Team 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout:Architecture: amd64 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout:Source: linux-meta 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout:Version: 5.15.0.105.102 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.3), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-105), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.3) 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout:Depends: linux-image-5.15.0-105-generic, linux-modules-extra-5.15.0-105-generic, linux-firmware, intel-microcode, amd64-microcode 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout:Recommends: thermald 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout:Description: Generic Linux kernel image 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout: This package will always depend on the latest generic kernel image 2024-04-25T06:54:06.414 INFO:teuthology.orchestra.run.smithi134.stdout: available. 2024-04-25T06:54:06.415 DEBUG:teuthology.orchestra.run.smithi134:> mktemp 2024-04-25T06:54:06.465 INFO:teuthology.orchestra.run.smithi134.stdout:/tmp/tmp.R9syjQ46aP 2024-04-25T06:54:06.466 DEBUG:teuthology.orchestra.run.smithi134:> sudo cp /boot/grub/grub.cfg /tmp/tmp.R9syjQ46aP 2024-04-25T06:54:06.526 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 0666 /tmp/tmp.R9syjQ46aP 2024-04-25T06:54:06.638 DEBUG:teuthology.orchestra.remote:smithi134:/tmp/tmp.R9syjQ46aP is 10KB 2024-04-25T06:54:06.651 DEBUG:teuthology.orchestra.run.smithi134:> rm -fr /tmp/tmp.R9syjQ46aP 2024-04-25T06:54:06.656 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-04-25T06:54:06.710 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T06:54:06.710 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-04-25T06:54:06.710 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-04-25T06:54:06.780 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-105-generic 2024-04-25T06:54:06.780 DEBUG:teuthology.orchestra.run.smithi134:> sudo update-grub 2024-04-25T06:54:08.017 INFO:teuthology.orchestra.run.smithi134.stderr:Sourcing file `/etc/default/grub' 2024-04-25T06:54:08.018 INFO:teuthology.orchestra.run.smithi134.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-04-25T06:54:08.021 INFO:teuthology.orchestra.run.smithi134.stderr:Generating grub configuration file ... 2024-04-25T06:54:08.269 INFO:teuthology.orchestra.run.smithi134.stderr:Found linux image: /boot/vmlinuz-5.15.0-105-generic 2024-04-25T06:54:08.275 INFO:teuthology.orchestra.run.smithi134.stderr:Found initrd image: /boot/initrd.img-5.15.0-105-generic 2024-04-25T06:54:08.566 INFO:teuthology.orchestra.run.smithi134.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-04-25T06:54:08.571 INFO:teuthology.orchestra.run.smithi134.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-04-25T06:54:08.613 INFO:teuthology.orchestra.run.smithi134.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-04-25T06:54:08.619 INFO:teuthology.orchestra.run.smithi134.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-04-25T06:54:08.750 INFO:teuthology.orchestra.run.smithi134.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-04-25T06:54:08.751 INFO:teuthology.orchestra.run.smithi134.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-04-25T06:54:08.751 INFO:teuthology.orchestra.run.smithi134.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-04-25T06:54:08.773 INFO:teuthology.orchestra.run.smithi134.stderr:done 2024-04-25T06:54:08.788 DEBUG:teuthology.orchestra.run.smithi134:> sudo shutdown -r now 2024-04-25T06:54:38.800 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-04-25T06:54:38.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-25T06:54:38.802 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:54:57.311 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-04-25T06:55:06.311 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-25T06:55:06.313 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:55:09.374 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-04-25T06:55:21.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-25T06:55:21.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:55:39.810 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.134 2024-04-25T06:55:54.818 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-25T06:55:54.819 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T06:55:55.216 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-04-25T06:55:55.838 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-25T06:55:55.838 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-105-generic"... 2024-04-25T06:55:55.838 DEBUG:teuthology.orchestra.run.smithi134:> uname -r 2024-04-25T06:55:55.883 INFO:teuthology.orchestra.run.smithi134.stdout:5.15.0-105-generic 2024-04-25T06:55:55.883 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-105-generic vs 5.15.0-105-generic 2024-04-25T06:55:55.883 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-25T06:55:55.884 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-04-25T06:55:56.884 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-04-25T06:55:56.884 DEBUG:teuthology.orchestra.run.smithi134:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-25T06:55:57.259 INFO:teuthology.orchestra.run.smithi134.stdout:ttyS1 2024-04-25T06:55:57.281 DEBUG:teuthology.parallel:result is None 2024-04-25T06:55:57.281 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-25T06:55:57.295 INFO:teuthology.task.internal:Creating test directory... 2024-04-25T06:55:57.295 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-25T06:55:57.300 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-25T06:55:57.372 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-25T06:55:57.381 INFO:teuthology.task.internal:Creating archive directory... 2024-04-25T06:55:57.381 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-25T06:55:57.404 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-25T06:55:57.411 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-25T06:55:57.411 DEBUG:teuthology.orchestra.run.smithi134:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-25T06:55:57.464 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:55:57.472 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T06:55:57.473 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-25T06:55:57.481 INFO:teuthology.task.internal:Configuring sudo... 2024-04-25T06:55:57.481 DEBUG:teuthology.orchestra.run.smithi134:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-25T06:55:57.538 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-25T06:55:57.547 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-25T06:55:57.547 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-25T06:55:57.584 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-25T06:55:57.632 DEBUG:teuthology.orchestra.run.smithi134:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-25T06:55:57.679 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T06:55:57.679 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-25T06:55:57.747 DEBUG:teuthology.orchestra.run.smithi134:> sudo service rsyslog restart 2024-04-25T06:55:57.872 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-25T06:55:57.880 INFO:teuthology.task.internal:Starting timer... 2024-04-25T06:55:57.881 INFO:teuthology.run_tasks:Running task pcp... 2024-04-25T06:55:57.890 INFO:teuthology.run_tasks:Running task selinux... 2024-04-25T06:55:57.898 DEBUG:teuthology.task.selinux:Excluding smithi134: OS 'ubuntu' does not support SELinux 2024-04-25T06:55:57.898 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-25T06:55:57.899 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-04-25T06:55:57.899 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-25T06:55:57.899 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-25T06:55:57.908 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-25T06:55:57.924 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-25T06:55:57.948 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi134.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-25T07:02:55.757 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi134.front.sepia.ceph.com')] 2024-04-25T07:02:55.758 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-25T07:02:55.759 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi134.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-25T07:02:55.842 DEBUG:teuthology.orchestra.run.smithi134:> true 2024-04-25T07:02:55.922 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi134.front.sepia.ceph.com' 2024-04-25T07:02:55.923 INFO:teuthology.run_tasks:Running task clock... 2024-04-25T07:02:55.933 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-25T07:02:55.933 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-25T07:02:55.934 DEBUG:teuthology.orchestra.run.smithi134:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T07:02:55.988 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-04-25T07:02:55.988 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: Command line: ntpd -gq 2024-04-25T07:02:55.989 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: ---------------------------------------------------- 2024-04-25T07:02:55.989 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: ntp-4 is maintained by Network Time Foundation, 2024-04-25T07:02:55.989 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-04-25T07:02:55.989 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: corporation. Support and training for ntp-4 are 2024-04-25T07:02:55.989 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: available at https://www.nwtime.org/support 2024-04-25T07:02:55.989 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: ---------------------------------------------------- 2024-04-25T07:02:55.989 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: proto: precision = 0.060 usec (-24) 2024-04-25T07:02:55.990 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: basedate set to 2022-02-04 2024-04-25T07:02:55.990 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: gps base set to 2022-02-06 (week 2196) 2024-04-25T07:02:55.990 INFO:teuthology.orchestra.run.smithi134.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T07:02:55.990 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-04-25T07:02:55.991 INFO:teuthology.orchestra.run.smithi134.stderr:restrict ::: KOD does nothing without LIMITED. 2024-04-25T07:02:55.992 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: restrict ::: KOD does nothing without LIMITED. 2024-04-25T07:02:55.992 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: Listen and drop on 0 v6wildcard [::]:123 2024-04-25T07:02:55.992 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-04-25T07:02:55.992 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: Listen normally on 2 lo 127.0.0.1:123 2024-04-25T07:02:55.992 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: Listen normally on 3 enp3s0f1 172.21.15.134:123 2024-04-25T07:02:55.992 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: Listen normally on 4 lo [::1]:123 2024-04-25T07:02:55.992 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6de9%5]:123 2024-04-25T07:02:55.992 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:02:55 ntpd[17220]: Listening on routing socket on fd #22 for interface updates 2024-04-25T07:02:56.989 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:56.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:56 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:57.989 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:57 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:57.989 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:57 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:58.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:58.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:58.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:58.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:02:58.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:02:58 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:00.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:03:00.992 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:03:00.992 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:00.992 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:00.992 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:03:00.992 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:00 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:03:02.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:02.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:02.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:02.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:02.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:02.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:02.990 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:02.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:02.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:03:02.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:03:02.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-04-25T07:03:02.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: can't open /var/log/ntpstats/rawstats.20240425: Permission denied 2024-04-25T07:03:02.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-04-25T07:03:02.991 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: can't open /var/log/ntpstats/peerstats.20240425: Permission denied 2024-04-25T07:03:02.991 INFO:teuthology.orchestra.run.smithi134.stdout:25 Apr 07:03:02 ntpd[17220]: ntpd: time slew -0.000735 s 2024-04-25T07:03:02.991 INFO:teuthology.orchestra.run.smithi134.stdout:ntpd: time slew -0.000735s 2024-04-25T07:03:02.992 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-04-25T07:03:02.992 INFO:teuthology.orchestra.run.smithi134.stderr:25 Apr 07:03:02 ntpd[17220]: can't open /var/log/ntpstats/loopstats.20240425: Permission denied 2024-04-25T07:03:03.103 INFO:teuthology.orchestra.run.smithi134.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T07:03:03.103 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================== 2024-04-25T07:03:03.103 INFO:teuthology.orchestra.run.smithi134.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:03:03.103 INFO:teuthology.orchestra.run.smithi134.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:03:03.103 INFO:teuthology.orchestra.run.smithi134.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:03:03.103 INFO:teuthology.orchestra.run.smithi134.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-04-25T07:03:03.104 INFO:teuthology.run_tasks:Running task install... 2024-04-25T07:03:03.113 DEBUG:teuthology.task.install:project ceph 2024-04-25T07:03:03.113 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'}} 2024-04-25T07:03:03.113 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f'} 2024-04-25T07:03:03.113 INFO:teuthology.task.install:Using flavor: default 2024-04-25T07:03:03.118 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-25T07:03:03.118 INFO:teuthology.task.install:extra packages: [] 2024-04-25T07:03:03.119 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-key list | grep Ceph 2024-04-25T07:03:03.204 INFO:teuthology.orchestra.run.smithi134.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-04-25T07:03:03.230 INFO:teuthology.orchestra.run.smithi134.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-04-25T07:03:03.230 INFO:teuthology.orchestra.run.smithi134.stdout:uid [ unknown] Ceph.com (release key) 2024-04-25T07:03:03.230 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-04-25T07:03:03.231 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T07:03:03.378 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default/ 2024-04-25T07:03:03.379 INFO:teuthology.task.install.deb:Package version is 19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:03:03.495 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T07:03:03.496 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-04-25T07:03:03.509 DEBUG:teuthology.orchestra.run.smithi134:> sudo apt-get update 2024-04-25T07:03:03.678 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-04-25T07:03:03.678 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-04-25T07:03:03.691 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-04-25T07:03:03.711 INFO:teuthology.orchestra.run.smithi134.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-04-25T07:03:03.726 INFO:teuthology.orchestra.run.smithi134.stdout:Ign:5 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy InRelease 2024-04-25T07:03:03.747 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-04-25T07:03:03.769 INFO:teuthology.orchestra.run.smithi134.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy Release.gpg 2024-04-25T07:03:03.790 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,426 B] 2024-04-25T07:03:03.817 INFO:teuthology.orchestra.run.smithi134.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-04-25T07:03:04.031 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 29.3 kB in 0s (86.0 kB/s) 2024-04-25T07:03:05.022 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-25T07:03:05.038 DEBUG:teuthology.orchestra.run.smithi134:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.0.0-3227-gc37f63b3-1jammy cephadm=19.0.0-3227-gc37f63b3-1jammy ceph-mds=19.0.0-3227-gc37f63b3-1jammy ceph-mgr=19.0.0-3227-gc37f63b3-1jammy ceph-common=19.0.0-3227-gc37f63b3-1jammy ceph-fuse=19.0.0-3227-gc37f63b3-1jammy ceph-test=19.0.0-3227-gc37f63b3-1jammy ceph-volume=19.0.0-3227-gc37f63b3-1jammy radosgw=19.0.0-3227-gc37f63b3-1jammy python3-rados=19.0.0-3227-gc37f63b3-1jammy python3-rgw=19.0.0-3227-gc37f63b3-1jammy python3-cephfs=19.0.0-3227-gc37f63b3-1jammy python3-rbd=19.0.0-3227-gc37f63b3-1jammy libcephfs2=19.0.0-3227-gc37f63b3-1jammy libcephfs-dev=19.0.0-3227-gc37f63b3-1jammy librados2=19.0.0-3227-gc37f63b3-1jammy librbd1=19.0.0-3227-gc37f63b3-1jammy rbd-fuse=19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:03:05.098 INFO:teuthology.orchestra.run.smithi134.stdout:Reading package lists... 2024-04-25T07:03:05.307 INFO:teuthology.orchestra.run.smithi134.stdout:Building dependency tree... 2024-04-25T07:03:05.308 INFO:teuthology.orchestra.run.smithi134.stdout:Reading state information... 2024-04-25T07:03:05.499 INFO:teuthology.orchestra.run.smithi134.stdout:The following packages were automatically installed and are no longer required: 2024-04-25T07:03:05.499 INFO:teuthology.orchestra.run.smithi134.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-04-25T07:03:05.499 INFO:teuthology.orchestra.run.smithi134.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-04-25T07:03:05.499 INFO:teuthology.orchestra.run.smithi134.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-04-25T07:03:05.500 INFO:teuthology.orchestra.run.smithi134.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-04-25T07:03:05.500 INFO:teuthology.orchestra.run.smithi134.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-04-25T07:03:05.500 INFO:teuthology.orchestra.run.smithi134.stdout:Use 'sudo apt autoremove' to remove them. 2024-04-25T07:03:05.501 INFO:teuthology.orchestra.run.smithi134.stdout:The following additional packages will be installed: 2024-04-25T07:03:05.501 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-04-25T07:03:05.501 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-04-25T07:03:05.501 INFO:teuthology.orchestra.run.smithi134.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-04-25T07:03:05.501 INFO:teuthology.orchestra.run.smithi134.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-04-25T07:03:05.502 INFO:teuthology.orchestra.run.smithi134.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-04-25T07:03:05.502 INFO:teuthology.orchestra.run.smithi134.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-04-25T07:03:05.503 INFO:teuthology.orchestra.run.smithi134.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: xmlstarlet zip 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout:Suggested packages: 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-04-25T07:03:05.504 INFO:teuthology.orchestra.run.smithi134.stdout:Recommended packages: 2024-04-25T07:03:05.505 INFO:teuthology.orchestra.run.smithi134.stdout: btrfs-tools 2024-04-25T07:03:05.603 INFO:teuthology.orchestra.run.smithi134.stdout:The following NEW packages will be installed: 2024-04-25T07:03:05.603 INFO:teuthology.orchestra.run.smithi134.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-04-25T07:03:05.603 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-04-25T07:03:05.603 INFO:teuthology.orchestra.run.smithi134.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-04-25T07:03:05.603 INFO:teuthology.orchestra.run.smithi134.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-04-25T07:03:05.603 INFO:teuthology.orchestra.run.smithi134.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-04-25T07:03:05.603 INFO:teuthology.orchestra.run.smithi134.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-04-25T07:03:05.604 INFO:teuthology.orchestra.run.smithi134.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-04-25T07:03:05.604 INFO:teuthology.orchestra.run.smithi134.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-04-25T07:03:05.604 INFO:teuthology.orchestra.run.smithi134.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-04-25T07:03:05.605 INFO:teuthology.orchestra.run.smithi134.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-04-25T07:03:05.606 INFO:teuthology.orchestra.run.smithi134.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-04-25T07:03:05.723 INFO:teuthology.orchestra.run.smithi134.stdout:0 upgraded, 118 newly installed, 0 to remove and 302 not upgraded. 2024-04-25T07:03:05.723 INFO:teuthology.orchestra.run.smithi134.stdout:Need to get 181 MB of archives. 2024-04-25T07:03:05.723 INFO:teuthology.orchestra.run.smithi134.stdout:After this operation, 829 MB of additional disk space will be used. 2024-04-25T07:03:05.723 INFO:teuthology.orchestra.run.smithi134.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-04-25T07:03:05.901 INFO:teuthology.orchestra.run.smithi134.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.0.0-3227-gc37f63b3-1jammy [3,556 kB] 2024-04-25T07:03:05.964 INFO:teuthology.orchestra.run.smithi134.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-04-25T07:03:05.971 INFO:teuthology.orchestra.run.smithi134.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-04-25T07:03:06.018 INFO:teuthology.orchestra.run.smithi134.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-04-25T07:03:06.223 INFO:teuthology.orchestra.run.smithi134.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-04-25T07:03:06.228 INFO:teuthology.orchestra.run.smithi134.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-04-25T07:03:06.244 INFO:teuthology.orchestra.run.smithi134.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-04-25T07:03:06.251 INFO:teuthology.orchestra.run.smithi134.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-04-25T07:03:06.251 INFO:teuthology.orchestra.run.smithi134.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-04-25T07:03:06.252 INFO:teuthology.orchestra.run.smithi134.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-04-25T07:03:06.264 INFO:teuthology.orchestra.run.smithi134.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-04-25T07:03:06.267 INFO:teuthology.orchestra.run.smithi134.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-04-25T07:03:06.270 INFO:teuthology.orchestra.run.smithi134.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-04-25T07:03:06.272 INFO:teuthology.orchestra.run.smithi134.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-04-25T07:03:06.305 INFO:teuthology.orchestra.run.smithi134.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-04-25T07:03:06.308 INFO:teuthology.orchestra.run.smithi134.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-04-25T07:03:06.312 INFO:teuthology.orchestra.run.smithi134.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-04-25T07:03:06.317 INFO:teuthology.orchestra.run.smithi134.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-04-25T07:03:06.318 INFO:teuthology.orchestra.run.smithi134.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-04-25T07:03:06.353 INFO:teuthology.orchestra.run.smithi134.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-04-25T07:03:06.354 INFO:teuthology.orchestra.run.smithi134.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-04-25T07:03:06.354 INFO:teuthology.orchestra.run.smithi134.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-04-25T07:03:06.355 INFO:teuthology.orchestra.run.smithi134.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-04-25T07:03:06.355 INFO:teuthology.orchestra.run.smithi134.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-04-25T07:03:06.397 INFO:teuthology.orchestra.run.smithi134.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-04-25T07:03:06.397 INFO:teuthology.orchestra.run.smithi134.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-04-25T07:03:06.398 INFO:teuthology.orchestra.run.smithi134.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-04-25T07:03:06.403 INFO:teuthology.orchestra.run.smithi134.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-04-25T07:03:06.404 INFO:teuthology.orchestra.run.smithi134.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-04-25T07:03:06.445 INFO:teuthology.orchestra.run.smithi134.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-04-25T07:03:06.445 INFO:teuthology.orchestra.run.smithi134.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-04-25T07:03:06.447 INFO:teuthology.orchestra.run.smithi134.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-04-25T07:03:06.447 INFO:teuthology.orchestra.run.smithi134.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-04-25T07:03:06.448 INFO:teuthology.orchestra.run.smithi134.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webob all 1:1.8.6-1.1 [85.2 kB] 2024-04-25T07:03:06.490 INFO:teuthology.orchestra.run.smithi134.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-04-25T07:03:06.491 INFO:teuthology.orchestra.run.smithi134.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-04-25T07:03:06.492 INFO:teuthology.orchestra.run.smithi134.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-04-25T07:03:06.502 INFO:teuthology.orchestra.run.smithi134.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-04-25T07:03:06.503 INFO:teuthology.orchestra.run.smithi134.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-04-25T07:03:06.533 INFO:teuthology.orchestra.run.smithi134.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-04-25T07:03:06.534 INFO:teuthology.orchestra.run.smithi134.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-04-25T07:03:06.536 INFO:teuthology.orchestra.run.smithi134.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-04-25T07:03:06.540 INFO:teuthology.orchestra.run.smithi134.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-idna all 3.3-1 [49.3 kB] 2024-04-25T07:03:06.542 INFO:teuthology.orchestra.run.smithi134.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-04-25T07:03:06.571 INFO:teuthology.orchestra.run.smithi134.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-04-25T07:03:06.573 INFO:teuthology.orchestra.run.smithi134.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.1 [181 kB] 2024-04-25T07:03:06.576 INFO:teuthology.orchestra.run.smithi134.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.0.0-3227-gc37f63b3-1jammy [3,224 kB] 2024-04-25T07:03:06.578 INFO:teuthology.orchestra.run.smithi134.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-04-25T07:03:06.579 INFO:teuthology.orchestra.run.smithi134.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-04-25T07:03:06.581 INFO:teuthology.orchestra.run.smithi134.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-04-25T07:03:06.615 INFO:teuthology.orchestra.run.smithi134.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-04-25T07:03:06.769 INFO:teuthology.orchestra.run.smithi134.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-04-25T07:03:06.771 INFO:teuthology.orchestra.run.smithi134.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.1 [108 kB] 2024-04-25T07:03:06.776 INFO:teuthology.orchestra.run.smithi134.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-04-25T07:03:06.777 INFO:teuthology.orchestra.run.smithi134.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-04-25T07:03:06.779 INFO:teuthology.orchestra.run.smithi134.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-04-25T07:03:06.821 INFO:teuthology.orchestra.run.smithi134.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-04-25T07:03:06.824 INFO:teuthology.orchestra.run.smithi134.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-04-25T07:03:06.825 INFO:teuthology.orchestra.run.smithi134.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-04-25T07:03:06.908 INFO:teuthology.orchestra.run.smithi134.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-04-25T07:03:06.908 INFO:teuthology.orchestra.run.smithi134.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-04-25T07:03:06.909 INFO:teuthology.orchestra.run.smithi134.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-04-25T07:03:06.910 INFO:teuthology.orchestra.run.smithi134.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-04-25T07:03:06.911 INFO:teuthology.orchestra.run.smithi134.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-04-25T07:03:06.912 INFO:teuthology.orchestra.run.smithi134.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-04-25T07:03:06.913 INFO:teuthology.orchestra.run.smithi134.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-04-25T07:03:06.913 INFO:teuthology.orchestra.run.smithi134.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-04-25T07:03:06.921 INFO:teuthology.orchestra.run.smithi134.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-04-25T07:03:06.933 INFO:teuthology.orchestra.run.smithi134.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-04-25T07:03:06.949 INFO:teuthology.orchestra.run.smithi134.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-04-25T07:03:06.950 INFO:teuthology.orchestra.run.smithi134.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-04-25T07:03:06.956 INFO:teuthology.orchestra.run.smithi134.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-04-25T07:03:06.961 INFO:teuthology.orchestra.run.smithi134.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-04-25T07:03:06.962 INFO:teuthology.orchestra.run.smithi134.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-04-25T07:03:06.963 INFO:teuthology.orchestra.run.smithi134.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.1 [474 kB] 2024-04-25T07:03:06.971 INFO:teuthology.orchestra.run.smithi134.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-04-25T07:03:06.973 INFO:teuthology.orchestra.run.smithi134.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-04-25T07:03:06.978 INFO:teuthology.orchestra.run.smithi134.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-04-25T07:03:06.994 INFO:teuthology.orchestra.run.smithi134.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-04-25T07:03:06.995 INFO:teuthology.orchestra.run.smithi134.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-04-25T07:03:06.996 INFO:teuthology.orchestra.run.smithi134.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-04-25T07:03:06.998 INFO:teuthology.orchestra.run.smithi134.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-04-25T07:03:06.999 INFO:teuthology.orchestra.run.smithi134.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-04-25T07:03:07.011 INFO:teuthology.orchestra.run.smithi134.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-04-25T07:03:07.020 INFO:teuthology.orchestra.run.smithi134.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-04-25T07:03:07.020 INFO:teuthology.orchestra.run.smithi134.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-04-25T07:03:07.024 INFO:teuthology.orchestra.run.smithi134.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-04-25T07:03:07.025 INFO:teuthology.orchestra.run.smithi134.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-04-25T07:03:07.134 INFO:teuthology.orchestra.run.smithi134.stdout:Get:90 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.0.0-3227-gc37f63b3-1jammy [712 kB] 2024-04-25T07:03:07.243 INFO:teuthology.orchestra.run.smithi134.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.0.0-3227-gc37f63b3-1jammy [356 kB] 2024-04-25T07:03:07.295 INFO:teuthology.orchestra.run.smithi134.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.0.0-3227-gc37f63b3-1jammy [32.8 kB] 2024-04-25T07:03:07.296 INFO:teuthology.orchestra.run.smithi134.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.0.0-3227-gc37f63b3-1jammy [180 kB] 2024-04-25T07:03:07.321 INFO:teuthology.orchestra.run.smithi134.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.0.0-3227-gc37f63b3-1jammy [67.1 kB] 2024-04-25T07:03:07.333 INFO:teuthology.orchestra.run.smithi134.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.0.0-3227-gc37f63b3-1jammy [330 kB] 2024-04-25T07:03:07.376 INFO:teuthology.orchestra.run.smithi134.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.0.0-3227-gc37f63b3-1jammy [6,893 kB] 2024-04-25T07:03:08.547 INFO:teuthology.orchestra.run.smithi134.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.0.0-3227-gc37f63b3-1jammy [112 kB] 2024-04-25T07:03:08.562 INFO:teuthology.orchestra.run.smithi134.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.0.0-3227-gc37f63b3-1jammy [466 kB] 2024-04-25T07:03:08.621 INFO:teuthology.orchestra.run.smithi134.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.0.0-3227-gc37f63b3-1jammy [25.8 MB] 2024-04-25T07:03:12.509 INFO:teuthology.orchestra.run.smithi134.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.0.0-3227-gc37f63b3-1jammy [5,092 kB] 2024-04-25T07:03:13.424 INFO:teuthology.orchestra.run.smithi134.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.0.0-3227-gc37f63b3-1jammy [244 kB] 2024-04-25T07:03:13.479 INFO:teuthology.orchestra.run.smithi134.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.0.0-3227-gc37f63b3-1jammy [124 kB] 2024-04-25T07:03:13.513 INFO:teuthology.orchestra.run.smithi134.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.0.0-3227-gc37f63b3-1jammy [1,526 kB] 2024-04-25T07:03:13.827 INFO:teuthology.orchestra.run.smithi134.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.0.0-3227-gc37f63b3-1jammy [6,257 kB] 2024-04-25T07:03:14.811 INFO:teuthology.orchestra.run.smithi134.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.0.0-3227-gc37f63b3-1jammy [22.8 MB] 2024-04-25T07:03:18.625 INFO:teuthology.orchestra.run.smithi134.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.0.0-3227-gc37f63b3-1jammy [14.1 kB] 2024-04-25T07:03:18.626 INFO:teuthology.orchestra.run.smithi134.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.0.0-3227-gc37f63b3-1jammy [870 kB] 2024-04-25T07:03:18.650 INFO:teuthology.orchestra.run.smithi134.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.0.0-3227-gc37f63b3-1jammy [2,437 kB] 2024-04-25T07:03:19.005 INFO:teuthology.orchestra.run.smithi134.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.0.0-3227-gc37f63b3-1jammy [791 kB] 2024-04-25T07:03:19.139 INFO:teuthology.orchestra.run.smithi134.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.0.0-3227-gc37f63b3-1jammy [148 kB] 2024-04-25T07:03:19.156 INFO:teuthology.orchestra.run.smithi134.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.0.0-3227-gc37f63b3-1jammy [2,297 kB] 2024-04-25T07:03:19.530 INFO:teuthology.orchestra.run.smithi134.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.0.0-3227-gc37f63b3-1jammy [8,625 kB] 2024-04-25T07:03:21.344 INFO:teuthology.orchestra.run.smithi134.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.0.0-3227-gc37f63b3-1jammy [14.2 kB] 2024-04-25T07:03:21.345 INFO:teuthology.orchestra.run.smithi134.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.0.0-3227-gc37f63b3-1jammy [51.6 MB] 2024-04-25T07:03:34.301 INFO:teuthology.orchestra.run.smithi134.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.0.0-3227-gc37f63b3-1jammy [124 kB] 2024-04-25T07:03:34.304 INFO:teuthology.orchestra.run.smithi134.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.0.0-3227-gc37f63b3-1jammy [39.0 kB] 2024-04-25T07:03:34.305 INFO:teuthology.orchestra.run.smithi134.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.0.0-3227-gc37f63b3-1jammy [13.6 MB] 2024-04-25T07:03:36.280 INFO:teuthology.orchestra.run.smithi134.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.0.0-3227-gc37f63b3-1jammy [91.4 kB] 2024-04-25T07:03:36.845 INFO:teuthology.orchestra.run.smithi134.stdout:Fetched 181 MB in 31s (5,905 kB/s) 2024-04-25T07:03:36.996 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-04-25T07:03:37.096 INFO:teuthology.orchestra.run.smithi134.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165995 files and directories currently installed.) 2024-04-25T07:03:37.102 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-04-25T07:03:37.178 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T07:03:37.506 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-04-25T07:03:37.532 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-04-25T07:03:37.573 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T07:03:37.934 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-04-25T07:03:37.960 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-04-25T07:03:38.000 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T07:03:38.437 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-04-25T07:03:38.463 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T07:03:38.511 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:03:39.432 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-04-25T07:03:39.459 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T07:03:39.708 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:03:39.985 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-04-25T07:03:40.011 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-04-25T07:03:40.052 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:03:40.438 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-04-25T07:03:40.464 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-04-25T07:03:40.504 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T07:03:40.825 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librados2. 2024-04-25T07:03:40.850 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../007-librados2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:40.891 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librados2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:41.344 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librbd1. 2024-04-25T07:03:41.370 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../008-librbd1_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:41.410 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librbd1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:42.105 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libcephfs2. 2024-04-25T07:03:42.131 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../009-libcephfs2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:42.179 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libcephfs2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:42.466 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rados. 2024-04-25T07:03:42.481 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../010-python3-rados_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:42.524 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rados (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:42.843 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-04-25T07:03:42.869 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:42.910 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-ceph-argparse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:43.154 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cephfs. 2024-04-25T07:03:43.180 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../012-python3-cephfs_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:43.229 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cephfs (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:43.615 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-ceph-common. 2024-04-25T07:03:43.641 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../013-python3-ceph-common_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:03:43.690 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:44.035 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-wcwidth. 2024-04-25T07:03:44.061 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-04-25T07:03:44.101 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T07:03:44.421 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-prettytable. 2024-04-25T07:03:44.447 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-04-25T07:03:44.487 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-04-25T07:03:44.757 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rbd. 2024-04-25T07:03:44.783 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../016-python3-rbd_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:44.831 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rbd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:45.209 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-04-25T07:03:45.236 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-04-25T07:03:45.276 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T07:03:45.537 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-04-25T07:03:45.552 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-04-25T07:03:45.611 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T07:03:45.883 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-04-25T07:03:45.899 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-04-25T07:03:45.941 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T07:03:46.286 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package lua5.1. 2024-04-25T07:03:46.313 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-04-25T07:03:46.353 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-04-25T07:03:46.647 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package lua-any. 2024-04-25T07:03:46.663 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-04-25T07:03:46.705 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking lua-any (27ubuntu1) ... 2024-04-25T07:03:46.941 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package zip. 2024-04-25T07:03:46.957 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-04-25T07:03:46.991 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking zip (3.0-12build2) ... 2024-04-25T07:03:47.261 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package unzip. 2024-04-25T07:03:47.288 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-04-25T07:03:47.328 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-04-25T07:03:47.740 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package luarocks. 2024-04-25T07:03:47.766 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-04-25T07:03:47.806 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-04-25T07:03:48.189 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package librgw2. 2024-04-25T07:03:48.216 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../025-librgw2_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:48.255 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking librgw2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:48.783 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rgw. 2024-04-25T07:03:48.798 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../026-python3-rgw_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:48.832 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:49.161 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package liboath0:amd64. 2024-04-25T07:03:49.187 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-04-25T07:03:49.227 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T07:03:49.580 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libradosstriper1. 2024-04-25T07:03:49.606 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../028-libradosstriper1_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:49.646 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libradosstriper1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:50.042 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-common. 2024-04-25T07:03:50.068 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../029-ceph-common_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:50.108 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:51.322 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-base. 2024-04-25T07:03:51.348 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../030-ceph-base_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:03:51.397 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-base (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:03:52.105 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-04-25T07:03:52.132 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-04-25T07:03:52.192 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-04-25T07:03:52.479 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cheroot. 2024-04-25T07:03:52.505 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-04-25T07:03:52.545 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T07:03:52.873 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-04-25T07:03:52.899 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-04-25T07:03:52.939 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-04-25T07:03:53.318 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jaraco.text. 2024-04-25T07:03:53.344 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-04-25T07:03:53.392 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-04-25T07:03:53.704 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-04-25T07:03:53.730 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-04-25T07:03:53.770 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-04-25T07:03:54.073 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-tempora. 2024-04-25T07:03:54.099 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-04-25T07:03:54.139 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-04-25T07:03:54.459 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-portend. 2024-04-25T07:03:54.485 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-04-25T07:03:54.534 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-04-25T07:03:54.862 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-04-25T07:03:54.888 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-04-25T07:03:54.928 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-04-25T07:03:55.231 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cherrypy3. 2024-04-25T07:03:55.257 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-04-25T07:03:55.297 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-04-25T07:03:55.626 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-natsort. 2024-04-25T07:03:55.642 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-04-25T07:03:55.676 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-04-25T07:03:55.980 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-logutils. 2024-04-25T07:03:56.006 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-04-25T07:03:56.046 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-04-25T07:03:56.349 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-markupsafe. 2024-04-25T07:03:56.375 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-04-25T07:03:56.432 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-04-25T07:03:56.735 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-mako. 2024-04-25T07:03:56.761 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-04-25T07:03:56.801 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T07:03:57.105 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-simplegeneric. 2024-04-25T07:03:57.131 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-04-25T07:03:57.163 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-04-25T07:03:57.466 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-singledispatch. 2024-04-25T07:03:57.492 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-04-25T07:03:57.532 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-04-25T07:03:57.877 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-webob. 2024-04-25T07:03:57.903 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1_all.deb ... 2024-04-25T07:03:57.951 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-webob (1:1.8.6-1.1) ... 2024-04-25T07:03:58.388 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-waitress. 2024-04-25T07:03:58.414 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-04-25T07:03:58.457 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T07:03:58.757 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-tempita. 2024-04-25T07:03:58.784 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-04-25T07:03:58.824 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T07:03:59.135 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-paste. 2024-04-25T07:03:59.162 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-04-25T07:03:59.202 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T07:03:59.521 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-04-25T07:03:59.536 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-04-25T07:03:59.579 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T07:03:59.882 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pastedeploy. 2024-04-25T07:03:59.909 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-04-25T07:03:59.949 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-04-25T07:04:00.252 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-webtest. 2024-04-25T07:04:00.278 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-04-25T07:04:00.318 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-04-25T07:04:00.605 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pecan. 2024-04-25T07:04:00.631 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-04-25T07:04:00.763 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T07:04:01.151 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-certifi. 2024-04-25T07:04:01.178 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-04-25T07:04:01.209 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-04-25T07:04:01.512 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-idna. 2024-04-25T07:04:01.539 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../055-python3-idna_3.3-1_all.deb ... 2024-04-25T07:04:01.579 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-idna (3.3-1) ... 2024-04-25T07:04:01.890 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-urllib3. 2024-04-25T07:04:01.916 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-04-25T07:04:01.956 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T07:04:02.276 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-requests. 2024-04-25T07:04:02.303 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-04-25T07:04:02.343 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T07:04:02.654 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-werkzeug. 2024-04-25T07:04:02.681 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.1_all.deb ... 2024-04-25T07:04:02.720 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T07:04:03.015 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-04-25T07:04:03.042 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:04:03.082 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-modules-core (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:03.428 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-04-25T07:04:03.455 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:03.495 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libsqlite3-mod-ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:03.814 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-bcrypt. 2024-04-25T07:04:03.841 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-04-25T07:04:03.880 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-04-25T07:04:04.250 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-openssl. 2024-04-25T07:04:04.276 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-04-25T07:04:04.317 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-04-25T07:04:04.728 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr. 2024-04-25T07:04:04.755 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../063-ceph-mgr_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:04.820 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:05.256 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mon. 2024-04-25T07:04:05.282 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../064-ceph-mon_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:05.322 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mon (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:05.918 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-osd. 2024-04-25T07:04:05.945 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../065-ceph-osd_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:05.984 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-osd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:07.096 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph. 2024-04-25T07:04:07.122 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../066-ceph_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:07.171 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:07.532 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-fuse. 2024-04-25T07:04:07.558 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../067-ceph-fuse_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:07.750 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:08.093 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mds. 2024-04-25T07:04:08.120 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../068-ceph-mds_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:08.159 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mds (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:08.512 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package cephadm. 2024-04-25T07:04:08.539 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../069-cephadm_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:08.579 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:08.915 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-asyncssh. 2024-04-25T07:04:08.942 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-04-25T07:04:08.981 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-04-25T07:04:09.301 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python-babel-localedata. 2024-04-25T07:04:09.327 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T07:04:09.359 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T07:04:10.407 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-babel. 2024-04-25T07:04:10.435 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-04-25T07:04:10.472 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T07:04:10.800 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-jinja2. 2024-04-25T07:04:10.827 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.1_all.deb ... 2024-04-25T07:04:10.866 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T07:04:11.220 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-04-25T07:04:11.247 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:04:11.286 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:11.623 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-repoze.lru. 2024-04-25T07:04:11.650 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-04-25T07:04:11.689 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-04-25T07:04:11.992 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-routes. 2024-04-25T07:04:12.019 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-04-25T07:04:12.058 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T07:04:12.395 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-04-25T07:04:12.422 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:04:12.486 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-dashboard (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:13.733 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-04-25T07:04:13.749 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-04-25T07:04:13.791 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T07:04:14.321 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-joblib. 2024-04-25T07:04:14.347 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-04-25T07:04:14.386 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T07:04:14.739 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-04-25T07:04:14.756 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-04-25T07:04:14.789 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-04-25T07:04:15.042 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-sklearn. 2024-04-25T07:04:15.069 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-04-25T07:04:15.108 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T07:04:15.800 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-04-25T07:04:15.828 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:04:15.953 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-diskprediction-local (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:16.964 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-cachetools. 2024-04-25T07:04:16.991 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-04-25T07:04:17.031 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-04-25T07:04:17.258 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pyasn1. 2024-04-25T07:04:17.271 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-04-25T07:04:17.300 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-04-25T07:04:17.620 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-04-25T07:04:17.647 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-04-25T07:04:17.686 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T07:04:17.989 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-rsa. 2024-04-25T07:04:18.017 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-04-25T07:04:18.061 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-rsa (4.8-1) ... 2024-04-25T07:04:18.560 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-google-auth. 2024-04-25T07:04:18.587 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-04-25T07:04:18.626 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-04-25T07:04:18.930 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-04-25T07:04:18.957 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-04-25T07:04:18.987 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T07:04:19.249 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-websocket. 2024-04-25T07:04:19.276 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-04-25T07:04:19.315 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-04-25T07:04:19.635 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-kubernetes. 2024-04-25T07:04:19.663 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-04-25T07:04:19.702 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T07:04:20.382 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-04-25T07:04:20.398 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:04:20.440 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-mgr-k8sevents (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:20.701 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libonig5:amd64. 2024-04-25T07:04:20.729 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-04-25T07:04:20.768 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T07:04:21.096 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libjq1:amd64. 2024-04-25T07:04:21.124 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T07:04:21.162 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T07:04:21.432 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package jq. 2024-04-25T07:04:21.459 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-04-25T07:04:21.498 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-04-25T07:04:21.751 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package socat. 2024-04-25T07:04:21.778 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-04-25T07:04:21.817 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-04-25T07:04:22.104 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package xmlstarlet. 2024-04-25T07:04:22.120 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-04-25T07:04:22.154 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-04-25T07:04:22.483 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-test. 2024-04-25T07:04:22.499 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../097-ceph-test_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:22.541 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-test (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:24.781 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package ceph-volume. 2024-04-25T07:04:24.808 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../098-ceph-volume_19.0.0-3227-gc37f63b3-1jammy_all.deb ... 2024-04-25T07:04:24.847 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking ceph-volume (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:25.143 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package libcephfs-dev. 2024-04-25T07:04:25.158 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../099-libcephfs-dev_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:25.201 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking libcephfs-dev (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:25.504 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package lua-socket:amd64. 2024-04-25T07:04:25.520 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-04-25T07:04:25.562 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T07:04:25.867 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package lua-sec:amd64. 2024-04-25T07:04:25.895 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-04-25T07:04:25.933 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-04-25T07:04:26.321 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package nvme-cli. 2024-04-25T07:04:26.348 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.1_amd64.deb ... 2024-04-25T07:04:26.387 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T07:04:26.809 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package pkg-config. 2024-04-25T07:04:26.837 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-04-25T07:04:26.875 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T07:04:27.095 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-04-25T07:04:27.111 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-04-25T07:04:27.153 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-04-25T07:04:27.573 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-iniconfig. 2024-04-25T07:04:27.589 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-04-25T07:04:27.631 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-04-25T07:04:27.926 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pastescript. 2024-04-25T07:04:27.953 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-04-25T07:04:27.992 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-04-25T07:04:28.329 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pluggy. 2024-04-25T07:04:28.357 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-04-25T07:04:28.395 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-04-25T07:04:28.707 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-psutil. 2024-04-25T07:04:28.734 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-04-25T07:04:28.773 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-04-25T07:04:29.085 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-py. 2024-04-25T07:04:29.113 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-04-25T07:04:29.151 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-py (1.10.0-1) ... 2024-04-25T07:04:29.463 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pygments. 2024-04-25T07:04:29.490 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-04-25T07:04:29.521 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T07:04:29.925 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pyinotify. 2024-04-25T07:04:29.941 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-04-25T07:04:29.983 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-04-25T07:04:30.294 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-toml. 2024-04-25T07:04:30.322 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-04-25T07:04:30.361 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-04-25T07:04:30.664 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-pytest. 2024-04-25T07:04:30.691 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-04-25T07:04:30.730 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T07:04:31.076 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package python3-simplejson. 2024-04-25T07:04:31.103 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-04-25T07:04:31.142 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-04-25T07:04:31.403 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-04-25T07:04:31.430 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-04-25T07:04:31.469 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-04-25T07:04:32.190 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package radosgw. 2024-04-25T07:04:32.218 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../116-radosgw_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:32.257 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking radosgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:33.076 INFO:teuthology.orchestra.run.smithi134.stdout:Selecting previously unselected package rbd-fuse. 2024-04-25T07:04:33.093 INFO:teuthology.orchestra.run.smithi134.stdout:Preparing to unpack .../117-rbd-fuse_19.0.0-3227-gc37f63b3-1jammy_amd64.deb ... 2024-04-25T07:04:33.135 INFO:teuthology.orchestra.run.smithi134.stdout:Unpacking rbd-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:33.457 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-04-25T07:04:33.898 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-04-25T07:04:34.024 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up nvme-cli (1.16-3ubuntu0.1) ... 2024-04-25T07:04:34.293 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-04-25T07:04:34.646 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-04-25T07:04:35.036 INFO:teuthology.orchestra.run.smithi134.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-04-25T07:04:35.109 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:35.685 INFO:teuthology.orchestra.run.smithi134.stdout:Adding system user cephadm....done 2024-04-25T07:04:35.816 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-04-25T07:04:36.084 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-04-25T07:04:36.335 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-04-25T07:04:36.461 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-04-25T07:04:36.729 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-04-25T07:04:36.989 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-04-25T07:04:37.115 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-py (1.10.0-1) ... 2024-04-25T07:04:37.408 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-04-25T07:04:37.768 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-04-25T07:04:38.037 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-04-25T07:04:38.329 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-04-25T07:04:38.573 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-04-25T07:04:38.824 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-04-25T07:04:39.074 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-04-25T07:04:39.357 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-ceph-argparse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:39.685 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-04-25T07:04:39.811 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-04-25T07:04:39.946 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-04-25T07:04:40.072 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-04-25T07:04:40.331 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-04-25T07:04:40.702 INFO:teuthology.orchestra.run.smithi134.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-04-25T07:04:40.729 INFO:teuthology.orchestra.run.smithi134.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-04-25T07:04:40.793 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-04-25T07:04:40.919 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-04-25T07:04:41.295 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-04-25T07:04:41.421 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-04-25T07:04:41.681 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-04-25T07:04:41.940 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-04-25T07:04:42.225 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up zip (3.0-12build2) ... 2024-04-25T07:04:42.342 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-04-25T07:04:43.036 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-04-25T07:04:43.321 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-04-25T07:04:43.447 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-04-25T07:04:43.573 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-04-25T07:04:43.857 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-04-25T07:04:44.242 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-04-25T07:04:44.493 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-04-25T07:04:44.878 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-04-25T07:04:45.188 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.1) ... 2024-04-25T07:04:45.547 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-04-25T07:04:45.798 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-04-25T07:04:45.941 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:46.251 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-idna (3.3-1) ... 2024-04-25T07:04:46.569 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-04-25T07:04:47.783 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-04-25T07:04:48.034 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:04:48.169 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-toml (0.10.2-1) ... 2024-04-25T07:04:48.495 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-04-25T07:04:48.646 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-04-25T07:04:48.781 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-04-25T07:04:49.074 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-04-25T07:04:49.325 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-04-25T07:04:49.576 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:04:49.702 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-04-25T07:04:50.029 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-04-25T07:04:50.288 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-04-25T07:04:50.564 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-04-25T07:04:50.815 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-04-25T07:04:51.092 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-04-25T07:04:51.351 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-04-25T07:04:51.477 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-04-25T07:04:51.761 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-04-25T07:04:51.896 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-04-25T07:04:52.189 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-webob (1:1.8.6-1.1) ... 2024-04-25T07:04:52.490 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-04-25T07:04:52.724 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-04-25T07:04:52.850 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-04-25T07:04:52.977 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-04-25T07:04:53.065 INFO:teuthology.orchestra.run.smithi134.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-04-25T07:04:53.312 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-04-25T07:04:53.438 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-04-25T07:04:53.831 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-04-25T07:04:54.082 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up lua-any (27ubuntu1) ... 2024-04-25T07:04:54.200 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-portend (3.0.0-1) ... 2024-04-25T07:04:54.451 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-04-25T07:04:54.577 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.1) ... 2024-04-25T07:04:54.928 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-04-25T07:04:55.246 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-04-25T07:04:55.522 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-04-25T07:04:55.651 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rsa (4.8-1) ... 2024-04-25T07:04:55.908 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-04-25T07:04:56.209 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-04-25T07:04:56.554 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-04-25T07:04:56.948 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-04-25T07:04:57.232 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-04-25T07:04:57.577 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-04-25T07:04:57.728 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librados2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:57.887 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libsqlite3-mod-ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:58.030 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-04-25T07:04:58.366 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libcephfs2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:58.492 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libradosstriper1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:58.644 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-04-25T07:04:58.937 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librbd1 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:59.080 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-modules-core (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:59.231 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:04:59.461 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T07:04:59.461 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-04-25T07:05:00.007 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up libcephfs-dev (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:00.134 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rados (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:00.259 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up librgw2 (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:00.386 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-04-25T07:05:01.598 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rbd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:01.740 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up rbd-fuse (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:01.867 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-rgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:01.993 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up python3-cephfs (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:02.119 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-common (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:02.746 INFO:teuthology.orchestra.run.smithi134.stdout:Adding group ceph....done 2024-04-25T07:05:03.122 INFO:teuthology.orchestra.run.smithi134.stdout:Adding system user ceph....done 2024-04-25T07:05:03.273 INFO:teuthology.orchestra.run.smithi134.stdout:Setting system user ceph properties....done 2024-04-25T07:05:03.288 INFO:teuthology.orchestra.run.smithi134.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-04-25T07:05:03.429 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-04-25T07:05:03.769 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-04-25T07:05:04.455 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-test (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:04.606 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up radosgw (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:05.149 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T07:05:05.150 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-04-25T07:05:05.616 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-base (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:05.965 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-04-25T07:05:06.426 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mds (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:06.650 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T07:05:06.650 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-04-25T07:05:07.161 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:07.396 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T07:05:07.396 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-04-25T07:05:07.904 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-osd (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:08.193 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T07:05:08.193 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-04-25T07:05:08.648 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-k8sevents (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:08.774 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-diskprediction-local (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:08.916 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mon (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:09.127 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T07:05:09.127 INFO:teuthology.orchestra.run.smithi134.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-04-25T07:05:09.577 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-cephadm (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:09.753 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:09.904 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-mgr-dashboard (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:10.166 INFO:teuthology.orchestra.run.smithi134.stdout:Setting up ceph-volume (19.0.0-3227-gc37f63b3-1jammy) ... 2024-04-25T07:05:10.509 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-04-25T07:05:10.618 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-04-25T07:05:10.779 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-04-25T07:05:11.922 INFO:teuthology.orchestra.run.smithi134.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-04-25T07:05:12.589 INFO:teuthology.orchestra.run.smithi134.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-04-25T07:05:12.596 DEBUG:teuthology.parallel:result is None 2024-04-25T07:05:12.596 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f 2024-04-25T07:05:12.740 DEBUG:teuthology.orchestra.run.smithi134:> dpkg-query -W -f '${Version}' ceph 2024-04-25T07:05:12.758 INFO:teuthology.orchestra.run.smithi134.stdout:19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:05:12.759 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3227-gc37f63b3-1jammy 2024-04-25T07:05:12.759 INFO:teuthology.task.install:The correct ceph version 19.0.0-3227-gc37f63b3-1jammy is installed. 2024-04-25T07:05:12.760 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-25T07:05:12.761 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T07:05:12.761 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-25T07:05:12.814 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-25T07:05:12.814 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T07:05:12.814 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/daemon-helper 2024-04-25T07:05:12.869 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-25T07:05:12.921 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-25T07:05:12.921 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T07:05:12.921 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-25T07:05:12.973 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-25T07:05:13.025 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-25T07:05:13.026 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T07:05:13.026 DEBUG:teuthology.orchestra.run.smithi134:> sudo dd of=/usr/bin/stdin-killer 2024-04-25T07:05:13.082 DEBUG:teuthology.orchestra.run.smithi134:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-25T07:05:13.137 INFO:teuthology.run_tasks:Running task workunit... 2024-04-25T07:05:13.161 INFO:tasks.workunit:Pulling workunits from ref a08304379f921e5e229e92c75ede8800059313ff 2024-04-25T07:05:13.162 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-25T07:05:13.162 INFO:tasks.workunit:timeout=3h 2024-04-25T07:05:13.162 INFO:tasks.workunit:cleanup=True 2024-04-25T07:05:13.162 DEBUG:teuthology.orchestra.run.smithi134:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-25T07:05:13.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T07:05:13.183 INFO:teuthology.orchestra.run.smithi134.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-04-25T07:05:13.183 DEBUG:teuthology.orchestra.run.smithi134:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T07:05:13.231 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-04-25T07:05:13.231 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-04-25T07:05:13.279 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/NitzanMordhai/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout a08304379f921e5e229e92c75ede8800059313ff 2024-04-25T07:05:13.329 INFO:tasks.workunit.client.0.smithi134.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-25T07:05:55.346 INFO:tasks.workunit.client.0.smithi134.stderr:Note: switching to 'a08304379f921e5e229e92c75ede8800059313ff'. 2024-04-25T07:05:55.346 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T07:05:55.346 INFO:tasks.workunit.client.0.smithi134.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr:state without impacting any branches by switching back to a branch. 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr: git switch -c 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr:Or undo this operation with: 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr: git switch - 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T07:05:55.347 INFO:tasks.workunit.client.0.smithi134.stderr:HEAD is now at a08304379f9 Merge pull request #56420 from rzarzynski/wip-tools-rados-get-with-offset 2024-04-25T07:05:55.349 DEBUG:teuthology.orchestra.run.smithi134:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-25T07:05:55.396 DEBUG:teuthology.orchestra.run.smithi134:> set -ex 2024-04-25T07:05:55.396 DEBUG:teuthology.orchestra.run.smithi134:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-25T07:05:55.442 INFO:tasks.workunit:Running workunits matching osd on client.0... 2024-04-25T07:05:55.442 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2024-04-25T07:05:55.442 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2024-04-25T07:05:55.505 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:05:55.538 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/bad-inc-map 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:55007 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:55007 2024-04-25T07:05:55.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2024-04-25T07:05:55.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2024-04-25T07:05:55.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none ' 2024-04-25T07:05:55.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:55007 ' 2024-04-25T07:05:55.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2024-04-25T07:05:55.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2024-04-25T07:05:55.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:05:55.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2024-04-25T07:05:55.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2024-04-25T07:05:55.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:05:55.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:05:55.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:05:55.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:05:55.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:05:55.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:05:55.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:05:55.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:05:55.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:05:55.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:55.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:05:55.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:05:55.547 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:05:55.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:05:55.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:05:55.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T07:05:55.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:05:55.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:55.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:05:55.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20269 2024-04-25T07:05:55.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:05:55.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:05:55.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2024-04-25T07:05:55.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:05:55.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:55.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:05:55.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.20269 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2024-04-25T07:05:55.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2024-04-25T07:05:56.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:05:56.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:56.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:56.158 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:56.159 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:56.159 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:05:56.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20269/$cluster-$name.asok' 2024-04-25T07:05:56.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:05:56.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:05:56.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:05:56.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:05:56.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:05:56.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:05:56.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:05:56.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:05:56.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:05:56.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:05:56.384 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:05:56.384 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:56.384 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:05:56.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20269/ceph-mon.a.asok 2024-04-25T07:05:56.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:05:56.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20269/ceph-mon.a.asok config get fsid 2024-04-25T07:05:56.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:05:56.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:05:56.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:05:56.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:05:56.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:05:56.498 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:05:56.498 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:05:56.498 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:05:56.498 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:05:56.498 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:56.498 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:05:56.498 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.20269/ceph-mon.a.asok 2024-04-25T07:05:56.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:05:56.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.20269/ceph-mon.a.asok config get mon_host 2024-04-25T07:05:56.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2024-04-25T07:05:56.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/bad-inc-map 2024-04-25T07:05:56.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:05:56.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:05:56.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:05:56.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/bad-inc-map/x 2024-04-25T07:05:56.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:05:57.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:05:57.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:57.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:57.546 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:57.546 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:57.546 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:05:57.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20269/$cluster-$name.asok' 2024-04-25T07:05:57.547 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:05:57.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/0 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none --mon-host=127.0.0.1:55007 ' 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:05:57.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20269/$cluster-$name.asok' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:05:57.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:05:57.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T07:05:57.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T07:05:57.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:05:57.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:05:57.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:05:57.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:05:57.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:05:57.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:05:57.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/0 2024-04-25T07:05:57.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:05:57.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=51813beb-1e82-48f0-87da-7e480f32ad41 2024-04-25T07:05:57.642 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 51813beb-1e82-48f0-87da-7e480f32ad41 2024-04-25T07:05:57.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 51813beb-1e82-48f0-87da-7e480f32ad41' 2024-04-25T07:05:57.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:05:57.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDVACpmefprJxAAZjE8ZOWBO+ftchGUh9BRPw== 2024-04-25T07:05:57.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDVACpmefprJxAAZjE8ZOWBO+ftchGUh9BRPw=="}' 2024-04-25T07:05:57.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 51813beb-1e82-48f0-87da-7e480f32ad41 -i td/bad-inc-map/0/new.json 2024-04-25T07:05:57.968 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:05:57.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/0/new.json 2024-04-25T07:05:57.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none --mon-host=127.0.0.1:55007 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVACpmefprJxAAZjE8ZOWBO+ftchGUh9BRPw== --osd-uuid 51813beb-1e82-48f0-87da-7e480f32ad41 2024-04-25T07:05:58.069 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:05:58.070+0000 7fe118b6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:58.072 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:05:58.074+0000 7fe118b6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:58.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:05:58.078+0000 7fe118b6e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:05:58.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:05:58.078+0000 7fe118b6e7c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T07:05:58.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:05:58.078+0000 7fe118b6e7c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T07:05:58.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:05:58.078+0000 7fe118b6e7c0 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2024-04-25T07:05:58.145 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:05:58.146+0000 7fe118b6e7c0 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2024-04-25T07:06:04.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/0/keyring 2024-04-25T07:06:04.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:06:04.444 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:06:04.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:06:04.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:06:04.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:06:04.684 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:06:04.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none --mon-host=127.0.0.1:55007 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:06:04.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:06:04.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:06:04.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:06:04.714 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:04.718+0000 7f17bc5447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:04.717 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:04.718+0000 7f17bc5447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:04.718 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:04.722+0000 7f17bc5447c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:04.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:06:04.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:06:04.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:06:04.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:06:04.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:06:04.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:04.911 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:06:04.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:06:04.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:04.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:05.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:05.305 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:05.306+0000 7f17bc5447c0 -1 Falling back to public interface 2024-04-25T07:06:06.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:06.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:06.146 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:06:06.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:06:06.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:06.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:06.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:07.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:07.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:07.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:06:07.386 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:06:07.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:07.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:07.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:08.085 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:08.089+0000 7f17bc5447c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:06:08.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:08.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:08.810 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:06:08.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:06:08.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:08.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:09.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:10.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:10.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:10.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:06:10.400 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:06:10.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:10.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:10.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:11.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:11.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:11.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:06:11.847 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:06:11.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:11.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:12.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:13.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:13.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:13.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:06:13.428 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:06:13.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:13.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:13.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:14.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:14.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:14.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:06:14.849 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:06:14.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:14.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:15.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:16.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:16.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:16.381 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:06:16.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:06:16.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:16.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:16.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:17.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:17.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:17.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:06:17.790 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:06:17.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:17.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:18.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:19.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:19.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:19.215 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:06:19.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:06:19.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:19.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:19.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:20.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:20.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:20.663 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:06:20.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:06:20.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:20.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:21.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:22.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:22.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:22.148 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:06:22.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:06:22.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:22.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:23.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:24.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:24.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:24.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:06:24.011 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:06:24.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:24.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:24.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:25.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:25.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:25.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:06:25.801 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:06:25.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:25.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:27.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:28.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:28.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:28.217 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:06:28.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:06:28.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:28.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:29.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:30.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:30.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:30.486 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:06:30.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:06:30.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:30.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:31.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:32.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:32.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:32.389 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:06:32.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:06:32.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:32.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:33.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:34.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:34.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:34.092 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:06:34.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:06:34.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:34.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:06:34.564 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3702791342,v1:127.0.0.1:6801/3702791342] [v2:127.0.0.1:6802/3702791342,v1:127.0.0.1:6803/3702791342] exists,up 51813beb-1e82-48f0-87da-7e480f32ad41 2024-04-25T07:06:34.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:06:34.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:06:34.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:06:34.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2024-04-25T07:06:34.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T07:06:34.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:06:34.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/1 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none --mon-host=127.0.0.1:55007 ' 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:06:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20269/$cluster-$name.asok' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/1 2024-04-25T07:06:34.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:06:34.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9a79d9c0-2590-4247-9342-9b342b079215 2024-04-25T07:06:34.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9a79d9c0-2590-4247-9342-9b342b079215' 2024-04-25T07:06:34.571 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 9a79d9c0-2590-4247-9342-9b342b079215 2024-04-25T07:06:34.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:06:34.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6ACpm5AP5IhAA4CAWQDSlS/ONtXVFfgp1QA== 2024-04-25T07:06:34.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6ACpm5AP5IhAA4CAWQDSlS/ONtXVFfgp1QA=="}' 2024-04-25T07:06:34.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9a79d9c0-2590-4247-9342-9b342b079215 -i td/bad-inc-map/1/new.json 2024-04-25T07:06:35.097 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:06:35.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/1/new.json 2024-04-25T07:06:35.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none --mon-host=127.0.0.1:55007 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6ACpm5AP5IhAA4CAWQDSlS/ONtXVFfgp1QA== --osd-uuid 9a79d9c0-2590-4247-9342-9b342b079215 2024-04-25T07:06:35.146 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:35.144+0000 7fb1a9e077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:35.161 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:35.160+0000 7fb1a9e077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:35.162 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:35.164+0000 7fb1a9e077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:35.162 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:35.164+0000 7fb1a9e077c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T07:06:35.163 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:35.164+0000 7fb1a9e077c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T07:06:35.163 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:35.164+0000 7fb1a9e077c0 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2024-04-25T07:06:35.192 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:35.192+0000 7fb1a9e077c0 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2024-04-25T07:06:41.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/1/keyring 2024-04-25T07:06:41.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:06:41.644 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:06:41.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:06:41.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:06:42.179 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:06:42.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:06:42.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:06:42.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:06:42.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none --mon-host=127.0.0.1:55007 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:06:42.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:06:42.209 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:42.207+0000 7fd42c87a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:42.230 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:42.227+0000 7fd42c87a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:42.231 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:42.231+0000 7fd42c87a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:06:42.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:06:42.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:06:42.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:06:42.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:06:42.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:06:42.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:42.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:06:42.636 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:06:42.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:42.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:42.805 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:42.803+0000 7fd42c87a7c0 -1 Falling back to public interface 2024-04-25T07:06:43.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:44.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:44.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:44.067 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:06:44.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:06:44.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:44.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:44.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:45.463 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:06:45.459+0000 7fd42c87a7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:06:45.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:45.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:45.484 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:06:45.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:06:45.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:45.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:46.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:47.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:47.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:47.137 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:06:47.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:06:47.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:47.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:47.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:48.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:48.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:48.616 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:06:48.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:06:48.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:48.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:49.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:50.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:50.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:50.100 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:06:50.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:06:50.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:50.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:50.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:51.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:51.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:51.527 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:06:51.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:06:51.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:51.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:51.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:52.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:52.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:52.939 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:06:52.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:06:52.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:52.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:53.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:54.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:54.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:54.422 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:06:54.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:06:54.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:54.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:55.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:56.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:56.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:56.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:06:56.097 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:06:56.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:56.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:56.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:57.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:57.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:57.558 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:06:57.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:06:57.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:57.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:57.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:06:58.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:06:58.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:06:58.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:06:58.971 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:06:58.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:06:58.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:06:59.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:00.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:00.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:00.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:07:00.770 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:07:00.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:00.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:07:01.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:02.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:02.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:02.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:07:02.673 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:07:02.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:02.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:07:03.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:04.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:04.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:04.387 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:07:04.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:07:04.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:04.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:07:04.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:05.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:05.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:05.868 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:07:05.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:07:05.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:05.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:07:06.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:07.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:07.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:07.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:07:07.612 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:07:07.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:07.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:07:08.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:07:09.681 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:07:09.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:09.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:07:10.134 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2900226559,v1:127.0.0.1:6811/2900226559] [v2:127.0.0.1:6812/2900226559,v1:127.0.0.1:6813/2900226559] exists,up 9a79d9c0-2590-4247-9342-9b342b079215 2024-04-25T07:07:10.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:10.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:10.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:10.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/bad-inc-map 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/bad-inc-map/2 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none --mon-host=127.0.0.1:55007 ' 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:07:10.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2024-04-25T07:07:10.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:07:10.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:07:10.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:07:10.138 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:07:10.138 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:07:10.138 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:07:10.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.20269/$cluster-$name.asok' 2024-04-25T07:07:10.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' 2024-04-25T07:07:10.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:07:10.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:07:10.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/bad-inc-map/2 2024-04-25T07:07:10.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:07:10.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=01358788-2f88-4882-a409-112a03964a3f 2024-04-25T07:07:10.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 01358788-2f88-4882-a409-112a03964a3f' 2024-04-25T07:07:10.141 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 01358788-2f88-4882-a409-112a03964a3f 2024-04-25T07:07:10.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:07:10.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeASpmiNiuCRAAtlh0UeAzxgxGXloDflwMQQ== 2024-04-25T07:07:10.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeASpmiNiuCRAAtlh0UeAzxgxGXloDflwMQQ=="}' 2024-04-25T07:07:10.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 01358788-2f88-4882-a409-112a03964a3f -i td/bad-inc-map/2/new.json 2024-04-25T07:07:10.788 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:07:10.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/bad-inc-map/2/new.json 2024-04-25T07:07:10.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none --mon-host=127.0.0.1:55007 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeASpmiNiuCRAAtlh0UeAzxgxGXloDflwMQQ== --osd-uuid 01358788-2f88-4882-a409-112a03964a3f 2024-04-25T07:07:10.837 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:10.834+0000 7efc8ef897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:10.844 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:10.842+0000 7efc8ef897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:10.846 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:10.842+0000 7efc8ef897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:10.846 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:10.842+0000 7efc8ef897c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T07:07:10.846 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:10.842+0000 7efc8ef897c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T07:07:10.846 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:10.842+0000 7efc8ef897c0 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2024-04-25T07:07:10.914 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:10.910+0000 7efc8ef897c0 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2024-04-25T07:07:17.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/bad-inc-map/2/keyring 2024-04-25T07:07:17.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:07:17.232 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:07:17.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:07:17.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:07:17.791 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:07:17.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:07:17.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b188f4f0-ba18-4968-bc28-7463df6af1e1 --auth-supported=none --mon-host=127.0.0.1:55007 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.20269/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:07:17.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:07:17.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:07:17.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:07:17.820 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:17.818+0000 7f30aab947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:17.822 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:17.818+0000 7f30aab947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:17.824 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:17.822+0000 7f30aab947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:07:18.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:07:18.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:07:18.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:07:18.232 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:07:18.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:07:18.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:07:18.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:18.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:07:18.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:18.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:18.382 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:18.378+0000 7f30aab947c0 -1 Falling back to public interface 2024-04-25T07:07:18.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:19.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:19.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:19.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:07:19.634 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:07:19.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:19.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:20.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:21.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:21.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:21.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:07:21.047 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:07:21.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:21.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:21.124 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:07:21.126+0000 7f30aab947c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:07:21.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:22.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:22.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:22.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:07:22.469 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:07:22.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:22.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:22.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:23.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:23.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:23.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:07:23.939 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:07:23.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:23.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:24.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:25.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:25.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:25.370 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:07:25.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:07:25.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:25.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:26.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:26.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:26.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:07:26.792 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:07:26.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:26.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:27.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:28.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:28.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:28.323 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:07:28.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:07:28.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:28.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:28.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:07:29.978 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:07:29.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:29.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:30.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:31.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:31.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:31.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:07:31.427 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:07:31.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:31.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:31.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:32.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:32.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:32.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:07:32.984 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:07:32.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:32.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:33.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:34.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:34.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:34.413 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:07:34.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:07:34.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:34.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:35.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:36.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:36.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:36.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:07:36.172 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:07:36.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:36.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:36.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:37.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:37.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:37.634 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:07:37.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:07:37.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:37.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:38.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:39.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:39.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:39.445 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:07:39.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:07:39.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:39.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:40.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:41.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:41.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:41.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:07:41.638 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:07:41.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:41.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:42.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:43.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:43.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:43.727 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:07:43.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:07:43.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:43.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:44.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:07:45.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:07:45.582 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:07:45.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:07:45.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:07:45.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:45.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:46.016 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/903692231,v1:127.0.0.1:6819/903692231] [v2:127.0.0.1:6820/903692231,v1:127.0.0.1:6821/903692231] exists,up 01358788-2f88-4882-a409-112a03964a3f 2024-04-25T07:07:46.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:46.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:46.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:46.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2024-04-25T07:07:46.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2024-04-25T07:07:46.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 8 2024-04-25T07:07:47.618 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T07:07:47.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:07:48.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2024-04-25T07:07:50.002 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 min_size to 1 2024-04-25T07:07:50.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2024-04-25T07:07:51.385 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 min_size to 2 2024-04-25T07:07:51.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2024-04-25T07:07:56.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T07:07:56.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2024-04-25T07:07:56.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:07:56.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:07:56.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:07:56.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:07:56.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T07:07:56.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:07:56.412 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:07:56.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:56.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:07:56.821 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3702791342,v1:127.0.0.1:6801/3702791342] [v2:127.0.0.1:6802/3702791342,v1:127.0.0.1:6803/3702791342] exists,up 51813beb-1e82-48f0-87da-7e480f32ad41 2024-04-25T07:07:56.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:07:56.823 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:07:56.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:56.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:07:57.267 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2900226559,v1:127.0.0.1:6811/2900226559] [v2:127.0.0.1:6812/2900226559,v1:127.0.0.1:6813/2900226559] exists,up 9a79d9c0-2590-4247-9342-9b342b079215 2024-04-25T07:07:57.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:57.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:57.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:57.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2024-04-25T07:07:57.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2024-04-25T07:07:57.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:07:57.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:07:57.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:07:57.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:07:57.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 10 )) 2024-04-25T07:07:57.269 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:07:57.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:07:57.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:07:57.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:07:57.685 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/903692231,v1:127.0.0.1:6819/903692231] [v2:127.0.0.1:6820/903692231,v1:127.0.0.1:6821/903692231] exists,up 01358788-2f88-4882-a409-112a03964a3f 2024-04-25T07:07:57.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:07:57.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:07:57.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:07:57.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2024-04-25T07:07:57.688 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:47.028+0000 7f309bb38640 -1 osd.2 15 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.688 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:47.616+0000 7f309bb38640 -1 osd.2 17 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.688 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:48.040+0000 7f309bb38640 -1 osd.2 18 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.688 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:49.572+0000 7f3099b34640 -1 osd.2 19 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.688 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:50.084+0000 7f3099b34640 -1 osd.2 23 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.688 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:50.572+0000 7f309bb38640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.688 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:50.572+0000 7f309bb38640 -1 osd.2 24 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.688 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:51.004+0000 7f309bb38640 -1 osd.2 25 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.688 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:51.612+0000 7f3099b34640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.689 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:51.716+0000 7f3099b34640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.689 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:51.820+0000 7f309bb38640 -1 osd.2 26 handle_osd_map injecting map crc failure 2024-04-25T07:07:57.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:47.028+0000 7f309bb38640 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:47.616+0000 7f309bb38640 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:48.040+0000 7f309bb38640 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:49.576+0000 7f3099b34640 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:50.084+0000 7f3099b34640 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:50.572+0000 7f309bb38640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:50.576+0000 7f309bb38640 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:51.008+0000 7f309bb38640 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:51.616+0000 7f3099b34640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:51.720+0000 7f3099b34640 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2024-04-25T07:07:57.691 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:07:51.824+0000 7f309bb38640 10 osd.2 26 handle_osd_map bailing because last < start (27<28) 2024-04-25T07:07:57.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2024-04-25T07:07:57.692 INFO:tasks.workunit.client.0.smithi134.stdout:success 2024-04-25T07:07:57.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2024-04-25T07:07:57.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=foo 2024-04-25T07:07:57.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2024-04-25T07:07:58.329 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' does not exist 2024-04-25T07:07:58.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2024-04-25T07:07:58.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:07:58.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:07:58.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:07:58.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:07:58.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:08:03.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:08:03.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2024-04-25T07:08:03.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T07:08:03.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:08:03.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T07:08:03.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:08:03.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:08:03.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:08:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:08:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:08:03.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:08:03.676 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:08:03.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:08:03.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:08:03.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:08:03.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:08:03.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:08:03.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:08:03.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:08:03.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:08:03.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:08:03.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:08:03.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:08:03.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T07:08:03.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:08:03.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:03.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:08:03.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20269 2024-04-25T07:08:03.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:08:03.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:08:03.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:08:03.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/bad-inc-map 0 2024-04-25T07:08:03.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2024-04-25T07:08:03.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:08:03.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2024-04-25T07:08:03.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:08:03.701 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:08:03.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:08:03.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:08:03.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:08:03.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:08:03.702 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:08:03.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:08:03.703 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:08:03.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:08:03.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:08:03.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:08:03.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:08:03.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:08:03.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:08:03.706 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:08:03.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:08:03.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:08:03.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2024-04-25T07:08:03.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:08:03.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:03.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.20269 2024-04-25T07:08:03.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.20269 2024-04-25T07:08:03.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:08:03.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:08:03.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:08:03.712 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:08:03.712 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:08:03.767 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2024-04-25T07:08:03.770 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2024-04-25T07:08:03.818 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/divergent-priors 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2024-04-25T07:08:03.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2024-04-25T07:08:03.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2024-04-25T07:08:03.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2024-04-25T07:08:03.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2024-04-25T07:08:03.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2024-04-25T07:08:03.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2024-04-25T07:08:03.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2024-04-25T07:08:03.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2024-04-25T07:08:03.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2024-04-25T07:08:03.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2024-04-25T07:08:03.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2024-04-25T07:08:03.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2024-04-25T07:08:03.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2024-04-25T07:08:03.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2024-04-25T07:08:03.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2024-04-25T07:08:03.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2024-04-25T07:08:03.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2024-04-25T07:08:03.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2024-04-25T07:08:03.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2024-04-25T07:08:03.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2024-04-25T07:08:03.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2024-04-25T07:08:03.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T07:08:03.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2024-04-25T07:08:03.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2024-04-25T07:08:03.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2024-04-25T07:08:03.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none ' 2024-04-25T07:08:03.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T07:08:03.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T07:08:03.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2024-04-25T07:08:03.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:03.832 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2024-04-25T07:08:03.832 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:08:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2024-04-25T07:08:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_divergent_2 2024-04-25T07:08:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_divergent_3 2024-04-25T07:08:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_divergent_ec' 2024-04-25T07:08:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:08:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:08:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:08:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:08:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:08:03.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:08:03.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:08:03.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:08:03.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:08:03.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:08:03.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:08:03.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:08:03.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:08:03.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:08:03.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:08:03.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:08:03.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:08:03.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:08:03.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:08:03.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:08:03.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:08:03.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:08:03.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:08:03.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:08:03.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:08:03.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:08:03.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:08:03.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:03.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:08:03.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26105 2024-04-25T07:08:03.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:08:03.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:08:03.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:08:03.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:08:03.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:03.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:08:03.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26105 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2024-04-25T07:08:03.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2024-04-25T07:08:03.851 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2024-04-25T07:08:03.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:08:03.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:08:04.337 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:08:04.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:04.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:04.338 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:04.338 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:04.338 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:08:04.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:08:04.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:08:04.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:08:04.567 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:08:04.567 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:08:04.567 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:08:04.567 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:08:04.567 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:08:04.568 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:08:04.568 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:08:04.568 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:08:04.568 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:08:04.569 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:04.569 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:08:04.569 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-mon.a.asok 2024-04-25T07:08:04.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:08:04.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26105/ceph-mon.a.asok config get fsid 2024-04-25T07:08:04.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:08:04.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:08:04.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:08:04.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:08:04.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:08:04.689 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:08:04.689 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:08:04.689 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:08:04.689 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:08:04.689 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:04.689 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:08:04.689 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-mon.a.asok 2024-04-25T07:08:04.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:08:04.690 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26105/ceph-mon.a.asok config get mon_host 2024-04-25T07:08:04.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2024-04-25T07:08:04.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:08:04.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:08:04.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:08:04.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:08:04.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:08:04.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:08:05.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:08:05.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:05.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:05.724 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:05.724 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:05.724 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:08:05.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:08:05.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:08:05.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:08:05.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:08:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:08:05.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2a90c41a-d91e-4c01-b1b8-9af4c2d95977 2024-04-25T07:08:05.802 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 2a90c41a-d91e-4c01-b1b8-9af4c2d95977 2024-04-25T07:08:05.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2a90c41a-d91e-4c01-b1b8-9af4c2d95977' 2024-04-25T07:08:05.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:08:05.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVASpmVO4mMRAA887bpojRtqR5aVbYhiO+JA== 2024-04-25T07:08:05.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVASpmVO4mMRAA887bpojRtqR5aVbYhiO+JA=="}' 2024-04-25T07:08:05.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2a90c41a-d91e-4c01-b1b8-9af4c2d95977 -i td/divergent-priors/0/new.json 2024-04-25T07:08:06.127 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:08:06.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:08:06.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBVASpmVO4mMRAA887bpojRtqR5aVbYhiO+JA== --osd-uuid 2a90c41a-d91e-4c01-b1b8-9af4c2d95977 2024-04-25T07:08:06.192 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:06.188+0000 7f9b6038b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:06.204 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:06.200+0000 7f9b6038b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:06.205 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:06.204+0000 7f9b6038b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:06.206 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:06.204+0000 7f9b6038b7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:08:06.206 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:06.204+0000 7f9b6038b7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:08:06.206 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:06.204+0000 7f9b6038b7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:08:06.236 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:06.232+0000 7f9b6038b7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:08:12.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:08:12.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:08:12.433 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:08:12.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:08:12.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:08:13.003 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:08:13.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:08:13.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:08:13.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:08:13.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:13.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:08:13.043 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:13.039+0000 7fb1682167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:13.052 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:13.047+0000 7fb1682167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:13.053 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:13.051+0000 7fb1682167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:13.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:08:13.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:13.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:08:13.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:13.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:13.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:13.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:13.500 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:08:13.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:13.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:13.634 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:13.631+0000 7fb1682167c0 -1 Falling back to public interface 2024-04-25T07:08:13.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:14.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:14.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:14.921 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:08:14.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:08:14.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:14.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:15.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:16.217 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:16.215+0000 7fb1682167c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:08:16.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:16.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:16.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:08:16.348 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:08:16.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:16.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:16.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:17.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:17.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:17.856 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:08:17.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:08:17.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:17.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:18.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:19.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:19.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:19.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:08:19.342 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:08:19.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:19.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:19.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:20.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:20.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:20.783 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:08:20.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:08:20.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:20.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:21.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:22.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:22.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:22.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:08:22.360 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:08:22.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:22.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:22.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:23.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:23.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:23.852 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:08:23.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:08:23.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:23.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:24.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:25.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:25.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:25.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:08:25.393 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:08:25.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:25.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:25.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:26.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:26.825 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:08:26.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:26.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:08:26.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:26.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:27.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:28.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:28.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:28.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:08:28.482 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:08:28.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:28.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:29.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:30.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:30.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:30.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:08:30.113 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:08:30.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:30.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:31.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:32.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:32.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:32.253 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:08:32.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:08:32.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:32.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:33.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:34.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:34.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:34.156 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:08:34.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:08:34.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:34.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:35.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:36.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:36.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:36.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:08:36.128 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:08:36.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:36.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:36.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:37.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:37.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:37.987 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:08:37.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:08:37.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:37.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:38.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:39.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:39.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:39.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:08:39.801 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:08:39.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:39.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:40.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:41.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:41.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:41.450 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:08:41.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:08:41.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:41.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:08:41.936 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/808983747,v1:127.0.0.1:6803/808983747] [v2:127.0.0.1:6804/808983747,v1:127.0.0.1:6805/808983747] exists,up 2a90c41a-d91e-4c01-b1b8-9af4c2d95977 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:08:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:08:41.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:08:41.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:08:41.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f198c0c1-66e3-4ae7-9073-15d722aaf43f 2024-04-25T07:08:41.943 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 f198c0c1-66e3-4ae7-9073-15d722aaf43f 2024-04-25T07:08:41.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f198c0c1-66e3-4ae7-9073-15d722aaf43f' 2024-04-25T07:08:41.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:08:41.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB5ASpmnATBORAAsntc6pLELNzx6U/KcG0KmA== 2024-04-25T07:08:41.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB5ASpmnATBORAAsntc6pLELNzx6U/KcG0KmA=="}' 2024-04-25T07:08:41.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f198c0c1-66e3-4ae7-9073-15d722aaf43f -i td/divergent-priors/1/new.json 2024-04-25T07:08:42.490 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:08:42.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:08:42.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB5ASpmnATBORAAsntc6pLELNzx6U/KcG0KmA== --osd-uuid f198c0c1-66e3-4ae7-9073-15d722aaf43f 2024-04-25T07:08:42.551 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:42.547+0000 7f3bae6a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:42.562 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:42.559+0000 7f3bae6a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:42.563 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:42.559+0000 7f3bae6a07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:42.564 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:42.559+0000 7f3bae6a07c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:08:42.564 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:42.559+0000 7f3bae6a07c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:08:42.564 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:42.559+0000 7f3bae6a07c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:08:42.663 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:42.659+0000 7f3bae6a07c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:08:49.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:08:49.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:08:49.190 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:08:49.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:08:49.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:08:49.770 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:08:49.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:08:49.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:08:49.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:08:49.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:08:49.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:08:49.810 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:49.807+0000 7f5462e907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:49.828 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:49.827+0000 7f5462e907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:49.830 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:49.827+0000 7f5462e907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:08:50.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:08:50.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:08:50.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:08:50.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:08:50.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:08:50.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:50.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:08:50.233 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:08:50.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:50.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:50.386 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:50.383+0000 7f5462e907c0 -1 Falling back to public interface 2024-04-25T07:08:50.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:51.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:51.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:51.665 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:08:51.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:08:51.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:51.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:52.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:53.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:53.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:53.107 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:08:53.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:08:53.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:53.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:53.163 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:08:53.159+0000 7f5462e907c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:08:53.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:54.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:54.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:54.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:08:54.695 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:08:54.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:54.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:55.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:56.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:56.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:56.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:08:56.137 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:08:56.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:56.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:56.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:57.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:57.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:57.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:08:57.553 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:08:57.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:57.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:58.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:08:59.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:08:59.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:08:59.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:08:59.028 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:08:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:08:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:08:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:00.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:00.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:00.498 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:09:00.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:09:00.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:00.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:01.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:02.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:02.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:02.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:09:02.030 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:09:02.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:02.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:02.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:03.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:03.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:03.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:09:03.574 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:09:03.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:03.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:04.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:05.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:05.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:05.148 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:09:05.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:09:05.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:05.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:05.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:06.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:06.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:06.597 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:09:06.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:09:06.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:06.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:07.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:08.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:08.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:08.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:09:08.095 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:09:08.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:08.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:09.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:10.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:10.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:10.063 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:09:10.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:09:10.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:10.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:10.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:11.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:11.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:11.850 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:09:11.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:09:11.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:11.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:12.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:13.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:13.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:13.731 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:09:13.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:09:13.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:13.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:14.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:15.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:15.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:15.555 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:09:15.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:09:15.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:15.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:16.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:17.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:17.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:17.377 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:09:17.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:09:17.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:17.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:17.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:18.851 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:09:18.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:09:18.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:18.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:09:19.282 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1750288021,v1:127.0.0.1:6811/1750288021] [v2:127.0.0.1:6812/1750288021,v1:127.0.0.1:6813/1750288021] exists,up f198c0c1-66e3-4ae7-9073-15d722aaf43f 2024-04-25T07:09:19.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:19.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:09:19.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:09:19.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:09:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:09:19.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:09:19.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:09:19.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=993b915e-c902-451f-aaba-3eb24893ac73 2024-04-25T07:09:19.290 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 993b915e-c902-451f-aaba-3eb24893ac73 2024-04-25T07:09:19.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 993b915e-c902-451f-aaba-3eb24893ac73' 2024-04-25T07:09:19.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:09:19.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfASpmV0XqEhAAJop1ZMJQldcdiN0jZlVSlQ== 2024-04-25T07:09:19.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfASpmV0XqEhAAJop1ZMJQldcdiN0jZlVSlQ=="}' 2024-04-25T07:09:19.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 993b915e-c902-451f-aaba-3eb24893ac73 -i td/divergent-priors/2/new.json 2024-04-25T07:09:19.828 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:09:19.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:09:19.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfASpmV0XqEhAAJop1ZMJQldcdiN0jZlVSlQ== --osd-uuid 993b915e-c902-451f-aaba-3eb24893ac73 2024-04-25T07:09:19.891 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:19.887+0000 7fe8c80667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:19.893 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:19.891+0000 7fe8c80667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:19.894 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:19.891+0000 7fe8c80667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:19.895 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:19.891+0000 7fe8c80667c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:09:19.895 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:19.891+0000 7fe8c80667c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:09:19.895 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:19.891+0000 7fe8c80667c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:09:19.935 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:19.931+0000 7fe8c80667c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:09:26.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:09:26.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:09:26.505 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:09:26.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:09:26.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:09:27.069 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:09:27.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:09:27.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:09:27.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:09:27.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:09:27.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:09:27.109 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:27.106+0000 7f545c52c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:27.121 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:27.118+0000 7f545c52c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:27.122 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:27.118+0000 7f545c52c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:09:27.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:09:27.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:09:27.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:09:27.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:09:27.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:09:27.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:27.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:09:27.522 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:09:27.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:27.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:27.685 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:27.682+0000 7f545c52c7c0 -1 Falling back to public interface 2024-04-25T07:09:27.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:28.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:28.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:28.981 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:09:28.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:09:28.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:28.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:29.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:30.283 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:09:30.278+0000 7f545c52c7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:09:30.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:30.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:30.409 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:09:30.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:09:30.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:30.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:30.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:31.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:31.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:31.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:09:31.966 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:09:31.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:31.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:32.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:33.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:33.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:33.387 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:09:33.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:09:33.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:33.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:33.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:34.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:34.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:34.844 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:09:34.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:09:34.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:34.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:35.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:36.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:36.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:36.343 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:09:36.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:09:36.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:36.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:36.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:37.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:37.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:37.781 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:09:37.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:09:37.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:37.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:38.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:39.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:39.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:39.328 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:09:39.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:09:39.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:39.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:39.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:40.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:40.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:40.825 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:09:40.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:09:40.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:40.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:41.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:42.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:42.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:42.291 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:09:42.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:09:42.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:42.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:42.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:43.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:43.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:43.808 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:09:43.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:09:43.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:43.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:44.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:45.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:45.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:45.280 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:09:45.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:09:45.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:45.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:46.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:47.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:47.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:47.087 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:09:47.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:09:47.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:47.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:48.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:49.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:49.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:49.132 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:09:49.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:09:49.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:49.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:49.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:50.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:50.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:50.922 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:09:50.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:09:50.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:50.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:51.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:52.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:52.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:52.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:09:52.725 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:09:52.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:52.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:53.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:54.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:54.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:54.552 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:09:54.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:09:54.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:54.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:55.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:09:56.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:09:56.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:09:56.074 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:09:56.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:09:56.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:09:56.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:09:56.566 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3601742442,v1:127.0.0.1:6819/3601742442] [v2:127.0.0.1:6820/3601742442,v1:127.0.0.1:6821/3601742442] exists,up 993b915e-c902-451f-aaba-3eb24893ac73 2024-04-25T07:09:56.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:09:56.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:09:56.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:09:56.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2024-04-25T07:09:57.293 INFO:tasks.workunit.client.0.smithi134.stderr:noout is set 2024-04-25T07:09:57.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2024-04-25T07:09:58.173 INFO:tasks.workunit.client.0.smithi134.stderr:noin is set 2024-04-25T07:09:58.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2024-04-25T07:09:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T07:09:59.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2024-04-25T07:09:59.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:09:59.603 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T07:09:59.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:10:00.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2024-04-25T07:10:01.379 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 3 2024-04-25T07:10:01.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2024-04-25T07:10:02.132 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 min_size to 2 2024-04-25T07:10:02.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2024-04-25T07:10:02.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:10:02.156 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:10:02.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:10:02.673 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:10:02.673 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:10:02.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:10:02.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:02.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:10:02.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836501 2024-04-25T07:10:02.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836501 2024-04-25T07:10:02.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501' 2024-04-25T07:10:02.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:02.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:10:03.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-04-25T07:10:03.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-04-25T07:10:03.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501 1-42949672975' 2024-04-25T07:10:03.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:03.035 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:10:03.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509451 2024-04-25T07:10:03.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509451 2024-04-25T07:10:03.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836501 1-42949672975 2-64424509451' 2024-04-25T07:10:03.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:03.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836501 2024-04-25T07:10:03.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:03.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:10:03.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836501 2024-04-25T07:10:03.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:03.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836501 2024-04-25T07:10:03.203 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836501 2024-04-25T07:10:03.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-04-25T07:10:03.203 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:10:03.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836499 -lt 21474836501 2024-04-25T07:10:03.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:10:04.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:10:04.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:10:05.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836501 2024-04-25T07:10:05.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:05.080 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-04-25T07:10:05.080 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:05.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:10:05.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-04-25T07:10:05.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:05.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-04-25T07:10:05.084 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672975 2024-04-25T07:10:05.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-25T07:10:05.085 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:10:05.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672975 2024-04-25T07:10:05.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:05.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:05.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509451 2024-04-25T07:10:05.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:10:05.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509451 2024-04-25T07:10:05.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:05.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509451 2024-04-25T07:10:05.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509451' 2024-04-25T07:10:05.507 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509451 2024-04-25T07:10:05.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:10:05.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509451 -lt 64424509451 2024-04-25T07:10:05.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2024-04-25T07:10:05.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:10:05.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:10:05.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:10:05.935 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:10:05.936 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:10:05.936 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:10:05.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:10:05.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:10:05.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:10:06.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:10:06.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:10:06.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:10:06.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:10:06.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:10:06.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:10:06.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:10:06.497 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:10:06.497 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:10:06.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:10:06.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:06.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:10:06.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-04-25T07:10:06.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-04-25T07:10:06.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-04-25T07:10:06.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:06.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:10:06.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T07:10:06.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T07:10:06.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978' 2024-04-25T07:10:06.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:06.832 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:10:07.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-04-25T07:10:07.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-04-25T07:10:07.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978 2-64424509454' 2024-04-25T07:10:07.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:07.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-04-25T07:10:07.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:07.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:10:07.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-04-25T07:10:07.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:07.037 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836504 2024-04-25T07:10:07.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-04-25T07:10:07.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-25T07:10:07.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:10:07.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836502 -lt 21474836504 2024-04-25T07:10:07.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:10:08.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:10:08.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:10:08.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836504 2024-04-25T07:10:08.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:08.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T07:10:08.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:08.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:10:08.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T07:10:08.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:08.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T07:10:08.872 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672978 2024-04-25T07:10:08.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T07:10:08.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:10:09.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-04-25T07:10:09.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:09.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-04-25T07:10:09.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:09.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:10:09.478 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-04-25T07:10:09.478 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:09.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-04-25T07:10:09.479 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509454 2024-04-25T07:10:09.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-25T07:10:09.480 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:10:09.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-25T07:10:09.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:10:09.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:09.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:10.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:10:10.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:10.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:10.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:10.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:10.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:10.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:10.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:10.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:10:10.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:10.899 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:10.899 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:11.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:10:11.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:10:11.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:10:11.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2024-04-25T07:10:11.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2024-04-25T07:10:11.867 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:10:11.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2024-04-25T07:10:11.885 INFO:tasks.workunit.client.0.smithi134.stdout:primary and soon to be divergent is 1 2024-04-25T07:10:11.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2024-04-25T07:10:11.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2024-04-25T07:10:12.290 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:12.290 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:10:00.704634+0000 0'0 29:19 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:25:00.509014+0000 0 0 2024-04-25T07:10:12.290 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:10:12.290 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:12.291 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:10:12.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2024-04-25T07:10:12.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T07:10:12.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2024-04-25T07:10:12.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2024-04-25T07:10:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T07:10:12.306 INFO:tasks.workunit.client.0.smithi134.stdout:writing initial objects 2024-04-25T07:10:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2024-04-25T07:10:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2024-04-25T07:10:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2024-04-25T07:10:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2024-04-25T07:10:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2024-04-25T07:10:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2024-04-25T07:10:12.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2024-04-25T07:10:12.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:12.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2024-04-25T07:10:12.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:12.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2024-04-25T07:10:12.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:12.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2024-04-25T07:10:12.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:12.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2024-04-25T07:10:12.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:12.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2024-04-25T07:10:13.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:13.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2024-04-25T07:10:13.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:13.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2024-04-25T07:10:13.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:13.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2024-04-25T07:10:13.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:13.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2024-04-25T07:10:13.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:13.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2024-04-25T07:10:14.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:14.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2024-04-25T07:10:14.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:14.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2024-04-25T07:10:14.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:14.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2024-04-25T07:10:14.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:14.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2024-04-25T07:10:14.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:14.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2024-04-25T07:10:14.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:14.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2024-04-25T07:10:15.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:15.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2024-04-25T07:10:15.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:15.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2024-04-25T07:10:15.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:15.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2024-04-25T07:10:15.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:15.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2024-04-25T07:10:15.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:15.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2024-04-25T07:10:15.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:15.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2024-04-25T07:10:16.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:16.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2024-04-25T07:10:16.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:16.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2024-04-25T07:10:16.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:16.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2024-04-25T07:10:16.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:16.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2024-04-25T07:10:16.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:16.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2024-04-25T07:10:16.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:16.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2024-04-25T07:10:17.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:17.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2024-04-25T07:10:17.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:17.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2024-04-25T07:10:17.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:17.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2024-04-25T07:10:17.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:17.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2024-04-25T07:10:17.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:17.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2024-04-25T07:10:17.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:17.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2024-04-25T07:10:18.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:18.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2024-04-25T07:10:18.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:18.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2024-04-25T07:10:18.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:18.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2024-04-25T07:10:18.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:18.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2024-04-25T07:10:18.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:18.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2024-04-25T07:10:18.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:18.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2024-04-25T07:10:19.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:19.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2024-04-25T07:10:19.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:19.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2024-04-25T07:10:19.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:19.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2024-04-25T07:10:19.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:19.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2024-04-25T07:10:19.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:19.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2024-04-25T07:10:19.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:19.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2024-04-25T07:10:20.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:20.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2024-04-25T07:10:20.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:20.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2024-04-25T07:10:20.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:20.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2024-04-25T07:10:20.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:20.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2024-04-25T07:10:20.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:20.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2024-04-25T07:10:21.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:21.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2024-04-25T07:10:21.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:21.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2024-04-25T07:10:21.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:21.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2024-04-25T07:10:21.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:21.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2024-04-25T07:10:21.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:21.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2024-04-25T07:10:21.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:21.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2024-04-25T07:10:22.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:22.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2024-04-25T07:10:22.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:22.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2024-04-25T07:10:22.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:22.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2024-04-25T07:10:22.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:22.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2024-04-25T07:10:22.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:22.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2024-04-25T07:10:22.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:22.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2024-04-25T07:10:23.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:23.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2024-04-25T07:10:23.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:23.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2024-04-25T07:10:23.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:23.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2024-04-25T07:10:23.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:23.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2024-04-25T07:10:23.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:23.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2024-04-25T07:10:23.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:23.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2024-04-25T07:10:24.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:24.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2024-04-25T07:10:24.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:24.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2024-04-25T07:10:24.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:24.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2024-04-25T07:10:24.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:24.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2024-04-25T07:10:24.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:24.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2024-04-25T07:10:25.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:25.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2024-04-25T07:10:25.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:25.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2024-04-25T07:10:25.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:25.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2024-04-25T07:10:25.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:25.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2024-04-25T07:10:25.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:25.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2024-04-25T07:10:25.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:25.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2024-04-25T07:10:26.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:26.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2024-04-25T07:10:26.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:26.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2024-04-25T07:10:26.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:26.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2024-04-25T07:10:26.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:26.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2024-04-25T07:10:26.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:26.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2024-04-25T07:10:26.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:26.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2024-04-25T07:10:27.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:27.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2024-04-25T07:10:27.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:27.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2024-04-25T07:10:27.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:27.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2024-04-25T07:10:27.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:27.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2024-04-25T07:10:27.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:27.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2024-04-25T07:10:27.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:27.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2024-04-25T07:10:28.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:28.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2024-04-25T07:10:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2024-04-25T07:10:28.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:28.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2024-04-25T07:10:28.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:28.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2024-04-25T07:10:28.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:28.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2024-04-25T07:10:29.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:29.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2024-04-25T07:10:29.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:29.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2024-04-25T07:10:29.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2024-04-25T07:10:29.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-04-25T07:10:29.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:10:29.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2024-04-25T07:10:29.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:10:29.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:10:29.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:10:29.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:10:29.556 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:10:29.556 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:10:29.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:10:29.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:10:29.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:10:29.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:10:29.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:10:29.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:10:29.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:10:29.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:10:29.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:10:30.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:10:30.053 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:10:30.053 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:10:30.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:10:30.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:30.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:10:30.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-04-25T07:10:30.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-04-25T07:10:30.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-04-25T07:10:30.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:30.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:10:30.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-04-25T07:10:30.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-04-25T07:10:30.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672985' 2024-04-25T07:10:30.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:10:30.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:10:30.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509460 2024-04-25T07:10:30.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509460 2024-04-25T07:10:30.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672985 2-64424509460' 2024-04-25T07:10:30.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:30.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-04-25T07:10:30.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:30.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:10:30.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-04-25T07:10:30.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:30.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-04-25T07:10:30.555 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836510 2024-04-25T07:10:30.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-04-25T07:10:30.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:10:30.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-04-25T07:10:30.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:30.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-04-25T07:10:30.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:30.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:10:30.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-04-25T07:10:30.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:30.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-04-25T07:10:30.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-25T07:10:30.984 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672985 2024-04-25T07:10:30.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:10:31.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672985 2024-04-25T07:10:31.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:10:32.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:10:32.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:10:32.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-04-25T07:10:32.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:10:32.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:10:32.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509460 2024-04-25T07:10:32.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:10:32.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-04-25T07:10:32.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:10:32.818 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509460 2024-04-25T07:10:32.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509460 2024-04-25T07:10:32.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-04-25T07:10:32.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:10:33.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509461 -lt 64424509460 2024-04-25T07:10:33.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:10:33.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:33.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:33.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:10:33.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:10:33.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:10:33.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:10:33.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:10:33.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:10:33.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:10:33.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:10:34.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:10:34.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:10:34.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:10:34.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:10:34.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:10:34.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:10:34.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:10:34.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2024-04-25T07:10:34.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:10:34.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:10:34.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:10:34.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:10:35.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2024-04-25T07:10:35.233 INFO:tasks.workunit.client.0.smithi134.stdout:blackholing osds 0 2 2024-04-25T07:10:35.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2024-04-25T07:10:35.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2024-04-25T07:10:35.627 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:35.628 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:10:00.704634+0000 29'100 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:25:00.509014+0000 0 0 2024-04-25T07:10:35.628 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:10:35.628 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:35.628 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:10:35.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:35.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2024-04-25T07:10:35.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:10:35.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:10:35.644 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:10:35.645 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:35.645 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:10:35.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.0.asok 2024-04-25T07:10:35.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-04-25T07:10:35.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:10:35.753 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:10:35.753 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:10:35.753 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:10:35.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:35.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2024-04-25T07:10:35.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:10:35.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:10:35.768 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:10:35.768 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:35.768 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:10:35.768 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.2.asok 2024-04-25T07:10:35.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2024-04-25T07:10:35.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:10:35.875 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:10:35.875 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:10:35.875 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:10:35.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2024-04-25T07:10:35.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2024-04-25T07:10:35.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2024-04-25T07:10:35.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2024-04-25T07:10:35.891 INFO:tasks.workunit.client.0.smithi134.stdout:writing divergent object 2024-04-25T07:10:35.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2024-04-25T07:10:35.891 INFO:tasks.workunit.client.0.smithi134.stdout:create missing divergent object 2024-04-25T07:10:35.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2024-04-25T07:10:35.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2024-04-25T07:10:35.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T07:10:35.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T07:10:35.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T07:10:35.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=existing_99 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/divergent-priors 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test existing_99 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-04-25T07:10:35.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=existing_99 2024-04-25T07:10:35.895 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T07:10:35.895 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test existing_99 2024-04-25T07:10:36.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-25T07:10:36.319 INFO:tasks.workunit.client.0.smithi134.stderr:0 2024-04-25T07:10:36.319 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:10:36.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 2024-04-25T07:10:36.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('1' '0' '2') 2024-04-25T07:10:36.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T07:10:36.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T07:10:36.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T07:10:36.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T07:10:36.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/divergent-priors/1/type 2024-04-25T07:10:36.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T07:10:36.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T07:10:36.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T07:10:36.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T07:10:36.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T07:10:36.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T07:10:36.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T07:10:36.322 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T07:10:36.322 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:10:36.322 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:10:36.322 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:10:36.322 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:36.322 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:10:36.323 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.1.asok 2024-04-25T07:10:36.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.26105/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T07:10:36.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T07:10:36.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T07:10:36.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T07:10:36.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T07:10:36.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:10:36.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:10:36.438 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:10:36.438 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:36.438 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:10:36.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.1.asok 2024-04-25T07:10:36.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T07:10:36.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.1.asok injectdataerr test existing_99 2024-04-25T07:10:36.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2024-04-25T07:10:36.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2024-04-25T07:10:46.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2024-04-25T07:10:46.572 INFO:tasks.workunit.client.0.smithi134.stderr:rados: no process found 2024-04-25T07:10:46.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2024-04-25T07:10:46.572 INFO:tasks.workunit.client.0.smithi134.stdout:killing all the osds 2024-04-25T07:10:46.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2024-04-25T07:10:46.983 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:46.983 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 active+clean 2024-04-25T07:10:36.734372+0000 29'101 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:25:00.509014+0000 0 0 2024-04-25T07:10:46.983 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:10:46.984 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:46.984 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:10:47.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:10:47.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:10:47.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:10:47.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:10:47.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:10:47.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:10:47.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:10:47.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T07:10:47.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2024-04-25T07:10:47.665 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:10:47.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T07:10:47.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2024-04-25T07:10:48.365 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already down. 2024-04-25T07:10:48.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2024-04-25T07:10:48.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2024-04-25T07:10:48.920 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already down. 2024-04-25T07:10:48.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:48.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2024-04-25T07:10:49.505 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T07:10:49.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:49.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2024-04-25T07:10:50.097 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already out. 2024-04-25T07:10:50.114 INFO:tasks.workunit.client.0.smithi134.stdout:bringing up non_divergent 0 2 2024-04-25T07:10:50.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2024-04-25T07:10:50.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2024-04-25T07:10:50.531 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:10:50.531 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 0 0 0 61700 0 0 17 84 17 stale+active+clean 2024-04-25T07:10:36.734372+0000 29'101 29:128 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:25:00.509014+0000 0 0 2024-04-25T07:10:50.531 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:10:50.531 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:10:50.531 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:10:50.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:50.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2024-04-25T07:10:50.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:10:50.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:10:50.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:10:50.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:10:50.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:50.549 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:10:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:10:50.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:10:50.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:10:50.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:10:50.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:10:50.552 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:10:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:10:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:10:50.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:10:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:10:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:10:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:10:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:10:50.591 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:50.594+0000 7f56adf2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:50.593 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:50.594+0000 7f56adf2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:50.594 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:50.594+0000 7f56adf2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:51.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:10:51.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:10:51.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:10:51.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:10:51.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:10:51.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:51.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:10:51.009 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:10:51.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:51.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:51.168 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:51.170+0000 7f56adf2b7c0 -1 Falling back to public interface 2024-04-25T07:10:51.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:52.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:52.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:52.469 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:10:52.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:10:52.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:52.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:52.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:53.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:53.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:53.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:10:53.897 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:10:53.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:53.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:54.130 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:54.130+0000 7f56adf2b7c0 -1 osd.0 29 log_to_monitors true 2024-04-25T07:10:54.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:55.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:55.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:55.350 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:10:55.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:10:55.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:55.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:10:55.786 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up out weight 0 up_from 36 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/405410448,v1:127.0.0.1:6803/405410448] [v2:127.0.0.1:6804/405410448,v1:127.0.0.1:6805/405410448] exists,up 2a90c41a-d91e-4c01-b1b8-9af4c2d95977 2024-04-25T07:10:55.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:10:55.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:10:55.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:10:55.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:10:55.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:10:55.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:10:55.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:10:55.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:10:55.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:10:55.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:10:55.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:10:55.791 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:10:55.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:10:55.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:10:55.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:10:55.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:10:55.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:10:55.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:10:55.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:10:55.828 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:55.830+0000 7ff5fa2097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:55.830 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:55.834+0000 7ff5fa2097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:55.832 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:55.834+0000 7ff5fa2097c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:10:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:10:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:10:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:10:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:10:56.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:10:56.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:56.221 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:10:56.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:10:56.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:56.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:56.400 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:56.402+0000 7ff5fa2097c0 -1 Falling back to public interface 2024-04-25T07:10:56.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:57.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:57.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:57.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:10:57.674 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:10:57.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:57.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:58.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:59.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:10:59.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:10:59.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:10:59.108 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:10:59.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:10:59.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:10:59.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:10:59.652 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:10:59.654+0000 7ff5fa2097c0 -1 osd.2 29 log_to_monitors true 2024-04-25T07:11:00.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:11:00.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:00.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:11:00.551 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:11:00.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:00.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:01.016 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up out weight 0 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/513351488,v1:127.0.0.1:6811/513351488] [v2:127.0.0.1:6812/513351488,v1:127.0.0.1:6813/513351488] exists,up 993b915e-c902-451f-aaba-3eb24893ac73 2024-04-25T07:11:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-04-25T07:11:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2024-04-25T07:11:01.557 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already in. 2024-04-25T07:11:01.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2024-04-25T07:11:01.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2024-04-25T07:11:02.088 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already in. 2024-04-25T07:11:02.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:11:02.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2024-04-25T07:11:02.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:11:02.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:11:02.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:11:02.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:11:02.106 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:11:02.106 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:11:02.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:11:02.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:11:02.107 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:11:02.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:11:02.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:11:02.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:11:02.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:11:02.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:11:02.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:11:02.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:11:02.678 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:11:02.679 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:11:02.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:11:02.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:02.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:11:02.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=154618822660 2024-04-25T07:11:02.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 154618822660 2024-04-25T07:11:02.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660' 2024-04-25T07:11:02.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:02.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:11:03.010 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:11:03.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:11:03.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:11:03.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:11:03.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:11:03.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:11:03.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757251 2024-04-25T07:11:03.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757251 2024-04-25T07:11:03.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-154618822660 2-163208757251' 2024-04-25T07:11:03.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:03.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:03.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-154618822660 2024-04-25T07:11:03.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:11:03.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-154618822660 2024-04-25T07:11:03.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:03.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=154618822660 2024-04-25T07:11:03.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 154618822660' 2024-04-25T07:11:03.181 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 154618822660 2024-04-25T07:11:03.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:11:03.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822658 -lt 154618822660 2024-04-25T07:11:03.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:11:04.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:11:04.661 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:11:05.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 154618822660 -lt 154618822660 2024-04-25T07:11:05.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:11:05.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-163208757251 2024-04-25T07:11:05.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:11:05.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:11:05.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-163208757251 2024-04-25T07:11:05.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:11:05.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757251 2024-04-25T07:11:05.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 163208757251' 2024-04-25T07:11:05.092 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 163208757251 2024-04-25T07:11:05.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:11:05.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757251 -lt 163208757251 2024-04-25T07:11:05.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:11:05.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:05.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:06.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:11:06.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:06.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:06.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:06.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:06.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:06.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:06.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:06.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:11:06.473 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:06.473 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:06.473 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:07.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:11:07.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:11:07.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:11:07.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:11:07.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:11:07.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:11:07.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:11:07.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:07.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:07.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:07.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:07.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:07.107 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:07.107 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:07.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:11:07.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:07.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:07.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:08.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:11:08.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:11:08.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:11:08.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:11:08.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:11:08.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:11:08.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:11:08.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:11:08.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:11:08.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:11:08.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:11:08.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:11:08.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:11:08.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:11:08.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:11:08.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:08.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:08.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:08.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:08.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:08.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:08.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:09.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:11:09.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:09.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:09.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:09.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:11:09.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:11:09.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:11:09.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:11:09.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:11:09.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:11:09.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:11:09.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:11:09.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:11:10.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:11:10.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:11:10.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:11:10.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:11:10.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:11:10.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:11:10.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:10.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:10.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:10.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:10.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:10.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:10.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:11.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:11:11.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:11.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:11.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:11.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:11:11.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:11:11.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:11:11.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:11:11.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:11:11.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:11:11.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:11:11.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:11:11.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:11:12.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:11:12.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:11:12.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:11:12.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:11:12.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:11:13.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:11:13.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:13.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:13.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:13.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:13.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:13.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:13.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:13.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:11:13.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:13.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:13.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:14.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:11:14.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:11:14.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:11:14.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:11:14.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:11:14.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:11:14.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:11:14.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:11:14.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:11:14.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:11:14.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:11:14.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:11:14.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:11:14.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:11:16.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:11:16.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:16.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:16.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:16.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:16.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:16.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:16.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:16.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:11:16.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:16.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:16.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:17.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:11:17.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:11:17.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:11:17.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:11:17.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:11:17.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:11:17.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:11:17.110 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:11:17.110 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:11:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:11:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:11:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:11:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:11:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:11:20.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:11:20.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:20.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:20.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:20.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:20.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:20.869 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:20.869 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:21.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:11:21.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:21.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:21.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:11:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:11:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:11:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:11:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:11:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:11:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:11:21.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:11:21.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:11:22.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:11:22.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:11:22.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:11:22.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:11:22.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:11:28.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:11:28.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:28.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:28.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:28.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:28.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:28.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:28.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:29.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:11:29.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:29.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:29.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:11:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:11:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:11:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:11:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:11:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:11:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:11:29.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:11:29.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:11:30.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:11:30.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:11:30.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:11:30.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:11:30.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:11:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:11:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:11:37.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:11:37.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:11:37.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:11:37.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:11:37.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:11:37.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:11:38.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:11:38.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:11:38.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:11:38.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:11:38.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:11:38.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:11:38.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:11:38.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:11:38.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:11:38.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:11:38.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:11:38.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:11:38.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:11:39.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:11:39.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:11:39.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:11:39.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:11:39.679 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:11:39.679 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_fingerprint": "9fea2449-91da-4540-a443-04151fcd1928", 2024-04-25T07:11:39.679 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:11:39.679 INFO:tasks.workunit.client.0.smithi134.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:11:39.679 INFO:tasks.workunit.client.0.smithi134.stdout: "timestamp": "2024-04-25T07:11:39.680423+0000", 2024-04-25T07:11:39.679 INFO:tasks.workunit.client.0.smithi134.stdout: "tag": "", 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "health": { 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "status": "HEALTH_WARN", 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "checks": { 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 17 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [], 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "OSD_DOWN": { 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:11:39.680 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 osds down", 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "osd.1 (root=default,host=smithi134) is down" 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "PG_DEGRADED": { 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:11:39.681 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.682 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: "mutes": [] 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_first_committed": 1, 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_last_committed": 1, 2024-04-25T07:11:39.683 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap": { 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:08:03.890624Z", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:08:03.890624Z", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release": 19, 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release_name": "squid", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "election_strategy": 1, 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "disallowed_leaders: ": "", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": false, 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "tiebreaker_mon": "", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_ranks: ": "", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "features": { 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "persistent": [ 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "kraken", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "luminous", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "mimic", 2024-04-25T07:11:39.684 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap-prune", 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "nautilus", 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "octopus", 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "pacific", 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "elector-pinging", 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "quincy", 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "reef", 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "squid" 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "optional": [] 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "mons": [ 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "rank": 0, 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "a", 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:11:39.685 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 0 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 0, 2024-04-25T07:11:39.686 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 0, 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_location": "{}" 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "quorum": [ 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: 0 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap": { 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 41, 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:08:04.916892+0000", 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:11:02.420002+0000", 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "last_up_change": "2024-04-25T07:10:59.962995+0000", 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "last_in_change": "2024-04-25T07:11:01.929168+0000", 2024-04-25T07:11:39.687 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_num": 5800384, 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_set": [ 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "nodown", 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "noin", 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "noout", 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "pglog_hardlimit", 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snapdirs", 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "recovery_deletes", 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "sortbitwise" 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_version": 9, 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:11:39.688 INFO:tasks.workunit.client.0.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_snapshot": "", 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_max": 1, 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "max_osd": 3, 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "min_compat_client": "jewel", 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "require_osd_release": "squid", 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "allow_crimson": false, 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "pools": [ 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_name": "test", 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "create_time": "2024-04-25T07:09:59.368311+0000", 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": 1, 2024-04-25T07:11:39.689 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_names": "hashpspool", 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 3, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "min_size": 2, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_rule": 0, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "object_hash": 2, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num": 1, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num": 1, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num_target": 1, 2024-04-25T07:11:39.690 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_target": 1, 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_pending": 1, 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pg_merge_meta": { 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "source_pgid": "0.0", 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "ready_epoch": 0, 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_started": 0, 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": 0, 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "source_version": "0'0", 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "target_version": "0'0" 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "last_change": "29", 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend": "0", 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "auid": 0, 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:11:39.691 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_seq": 0, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_epoch": 0, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_snaps": [], 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps": "[]", 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_bytes": 0, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_objects": 0, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "tiers": [], 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "tier_of": -1, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "read_tier": -1, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "write_tier": -1, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_mode": "none", 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_bytes": 0, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_objects": 0, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:11:39.692 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_flush_age": 0, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_evict_age": 0, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profile": "", 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_params": { 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "none" 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_period": 0, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_count": 0, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "use_gmt_hitset": true, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "grade_table": [], 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "stripe_width": 0, 2024-04-25T07:11:39.693 INFO:tasks.workunit.client.0.smithi134.stdout: "expected_num_objects": 0, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "fast_read": false, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "options": {}, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "application_metadata": {}, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "read_balance": { 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "score_type": "Fair distribution", 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "score_acting": 2.9900000095367432, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "score_stable": 2.9900000095367432, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_acting": 2, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_stable": 2, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:11:39.694 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity": 1, 2024-04-25T07:11:39.695 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:11:39.695 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.695 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.695 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.695 INFO:tasks.workunit.client.0.smithi134.stdout: "osds": [ 2024-04-25T07:11:39.695 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.695 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:11:39.695 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "2a90c41a-d91e-4c01-b1b8-9af4c2d95977", 2024-04-25T07:11:39.695 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 5, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 29, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 36, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 40, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 30, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.696 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 405410448 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 405410448 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.697 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 405410448 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 405410448 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.698 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 405410448 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 405410448 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.699 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 405410448 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 405410448 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6803/405410448", 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6805/405410448", 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6809/405410448", 2024-04-25T07:11:39.700 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6807/405410448", 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "f198c0c1-66e3-4ae7-9073-15d722aaf43f", 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 0, 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T07:11:39.701 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 10, 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 23, 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 31, 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1750288021 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1750288021 2024-04-25T07:11:39.702 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1750288021 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1750288021 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.703 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1750288021 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1750288021 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.704 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1750288021 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1750288021 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/1750288021", 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/1750288021", 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/1750288021", 2024-04-25T07:11:39.705 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/1750288021", 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "exists" 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "993b915e-c902-451f-aaba-3eb24893ac73", 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 15, 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 29, 2024-04-25T07:11:39.706 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 38, 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 0, 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 32, 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 513351488 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:11:39.707 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 513351488 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 513351488 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 513351488 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.708 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 513351488 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 513351488 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.709 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 513351488 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 513351488 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/513351488", 2024-04-25T07:11:39.710 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/513351488", 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/513351488", 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/513351488", 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_xinfo": [ 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:10:47.446806+0000", 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:11:39.711 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:08:39.954682+0000", 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:10:48.091773+0000", 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:09:17.319037+0000", 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:11:39.712 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:10:48.746627+0000", 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:09:54.796844+0000", 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap": [], 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_items": [], 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_primaries": [], 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_temp": [], 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_temp": [], 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "blocklist": {}, 2024-04-25T07:11:39.713 INFO:tasks.workunit.client.0.smithi134.stdout: "range_blocklist": {}, 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profiles": { 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "default": { 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "2", 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van" 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps_queue": [], 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "new_removed_snaps": [], 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "new_purged_snaps": [], 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_node_flags": {}, 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "device_class_flags": {}, 2024-04-25T07:11:39.714 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": { 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_enabled": false, 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_bucket_count": 0, 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_bucket": 0 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_metadata": [ 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6804/405410448,v1:127.0.0.1:6805/405410448]", 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:11:39.715 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:11:39.716 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:08:11.758566Z", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:11:39.717 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6802/405410448,v1:127.0.0.1:6803/405410448]", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/405410448,v1:127.0.0.1:6809/405410448]", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/405410448,v1:127.0.0.1:6807/405410448]", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:11:39.718 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/1750288021,v1:127.0.0.1:6813/1750288021]", 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:11:39.719 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:11:39.720 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:08:48.501808Z", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/1750288021,v1:127.0.0.1:6811/1750288021]", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:11:39.721 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/1750288021,v1:127.0.0.1:6817/1750288021]", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/1750288021,v1:127.0.0.1:6815/1750288021]", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:11:39.722 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/513351488,v1:127.0.0.1:6813/513351488]", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:11:39.723 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:11:39.724 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:09:25.810084Z", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/513351488,v1:127.0.0.1:6811/513351488]", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/513351488,v1:127.0.0.1:6817/513351488]", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/513351488,v1:127.0.0.1:6815/513351488]", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:11:39.725 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.726 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_clean_epochs": { 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": { 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: "per_pool": [] 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_epochs": [ 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 36 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 38 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.727 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_first_committed": 1, 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_last_committed": 41, 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "crushmap": { 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": [ 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.0", 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.1", 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:11:39.728 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.2", 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: "types": [ 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 0, 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd" 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "host" 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 2, 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "chassis" 2024-04-25T07:11:39.729 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 3, 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "rack" 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 4, 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "row" 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 5, 2024-04-25T07:11:39.730 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pdu" 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 6, 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pod" 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 7, 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "room" 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 8, 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "datacenter" 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.731 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 9, 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "zone" 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 10, 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "region" 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "root" 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: "buckets": [ 2024-04-25T07:11:39.732 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -1, 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default", 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.733 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -2, 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default~hdd", 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.734 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134", 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:11:39.735 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134~hdd", 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:11:39.736 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.737 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "rules": [ 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 0, 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "replicated_rule", 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_firstn", 2024-04-25T07:11:39.738 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "tunables": { 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_tries": 0, 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_total_tries": 50, 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_stable": 1, 2024-04-25T07:11:39.739 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_descents": 100, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_collision_tries": 100, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "straw_calc_version": 1, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "profile": "jewel", 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_tunables": 1, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "legacy_tunables": 0, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "minimum_required_version": "jewel", 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables": 1, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables2": 1, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v2_rules": 0, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables3": 1, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v3_rules": 0, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v4_buckets": 1, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables5": 1, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v5_rules": 0, 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: "has_msr_rules": 0 2024-04-25T07:11:39.740 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_args": {} 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "fsmap": { 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "default_fscid": -1, 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": { 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": {}, 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "ro_compat": {}, 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "incompat": { 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_1": "base v0.20", 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:11:39.741 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_8": "no anchor table", 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_9": "file layout v2", 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_11": "minor log segments", 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_flags": { 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "enable_multiple": true, 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "ever_enabled_multiple": true 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "standbys": [], 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "filesystems": [] 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:11:39.742 INFO:tasks.workunit.client.0.smithi134.stdout: "auth": { 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 5, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_secrets": 3 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_active": 1, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_unknown": 0, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osd": 3, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_sum": { 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 61700, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:11:39.743 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 100, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 100, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:11:39.744 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:11:39.745 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:11:39.746 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 16, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 16, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 2, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 2, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 0 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum": { 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:11:39.747 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83624, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2112, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314489176, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036916224, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2162688, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 460116, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:11:39.748 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 1, 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:11:39.749 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum_by_class": { 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "hdd": { 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83624, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2112, 2024-04-25T07:11:39.750 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314489176, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036916224, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2162688, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 460116, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.751 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 1, 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:11:39.752 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_stats": [ 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 61700, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:11:39.753 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 100, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 100, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:11:39.754 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:11:39.755 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 1228800, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 185100, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:11:39.756 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 16, 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 16, 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 2, 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 2, 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 3 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_stats": [ 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:11:39.757 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 154618822667 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 42949672988 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 163208757258 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_state": [ 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.758 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "active+undersized+degraded", 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 1 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_osd": [ 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 1, 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:11:39.759 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 0, 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": [ 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": {} 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: "servicemap": { 2024-04-25T07:11:39.760 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 5, 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:10:50.302013+0000", 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "services": {} 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_last_committed": 96, 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_first_committed": 1, 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_last_committed": 388, 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "paxos": { 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 533, 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pn": 0, 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: "accepted_pn": 0 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:11:39.761 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:11:39.762 INFO:tasks.workunit.client.0.smithi134.stderr:report 68786140 2024-04-25T07:11:39.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:11:39.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2024-04-25T07:11:39.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2024-04-25T07:11:39.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2024-04-25T07:11:39.763 INFO:tasks.workunit.client.0.smithi134.stdout:writing non-divergent object existing_4 2024-04-25T07:11:39.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:40.085 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:40.085 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:11:02.789422+0000 29'100 41:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:17:21.139669+0000 0 0 2024-04-25T07:11:40.085 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:11:40.085 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:40.085 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:11:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:11:40.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2024-04-25T07:11:40.215 INFO:tasks.workunit.client.0.smithi134.stdout:delay recovery 2024-04-25T07:11:40.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:40.612 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:40.613 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:11:02.789422+0000 29'100 41:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:17:21.139669+0000 0 0 2024-04-25T07:11:40.613 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:11:40.613 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:40.613 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:11:40.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-04-25T07:11:40.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2024-04-25T07:11:40.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:11:40.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:11:40.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:11:40.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:40.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:11:40.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.0.asok 2024-04-25T07:11:40.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-04-25T07:11:40.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:11:40.729 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:11:40.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2024-04-25T07:11:40.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2024-04-25T07:11:40.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:11:40.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:11:40.742 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:11:40.742 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:40.742 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:11:40.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.2.asok 2024-04-25T07:11:40.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2024-04-25T07:11:40.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:11:40.856 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:11:40.868 INFO:tasks.workunit.client.0.smithi134.stdout:revive divergent 1 2024-04-25T07:11:40.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2024-04-25T07:11:40.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:41.281 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:41.281 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:11:02.789422+0000 29'100 41:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:17:21.139669+0000 0 0 2024-04-25T07:11:41.281 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:11:41.281 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:41.281 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:11:41.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2024-04-25T07:11:41.981 INFO:tasks.workunit.client.0.smithi134.stderr:noup is set 2024-04-25T07:11:42.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2024-04-25T07:11:42.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:11:42.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:11:42.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:11:42.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:11:42.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:11:42.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:42.005 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:11:42.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:11:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:11:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:11:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:11:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:11:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:11:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:11:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:11:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:11:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:11:42.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:11:42.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:11:42.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:11:42.008 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:11:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:11:42.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:11:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:11:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:11:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:11:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:11:42.047 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:42.049+0000 7f53be46b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:42.057 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:42.057+0000 7f53be46b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:42.058 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:42.061+0000 7f53be46b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:42.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:11:42.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2024-04-25T07:11:42.593 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:42.593+0000 7f53be46b7c0 -1 Falling back to public interface 2024-04-25T07:11:45.609 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:45.609+0000 7f53be46b7c0 -1 osd.1 29 log_to_monitors true 2024-04-25T07:11:47.448 INFO:tasks.workunit.client.0.smithi134.stdout:delay recovery divergent 2024-04-25T07:11:47.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2024-04-25T07:11:47.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:47.857 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:47.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:11:02.789422+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:17:21.139669+0000 0 0 2024-04-25T07:11:47.858 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:11:47.858 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:47.858 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:11:47.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2024-04-25T07:11:47.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:11:47.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:11:47.874 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:11:47.874 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:47.874 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:11:47.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.1.asok 2024-04-25T07:11:47.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2024-04-25T07:11:47.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:11:47.986 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:11:47.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2024-04-25T07:11:48.661 INFO:tasks.workunit.client.0.smithi134.stderr:noup is unset 2024-04-25T07:11:48.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2024-04-25T07:11:48.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:48.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:11:48.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:48.682 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:11:48.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:48.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:48.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:48.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:48.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:11:49.265 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 36 up_thru 40 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/405410448,v1:127.0.0.1:6803/405410448] [v2:127.0.0.1:6804/405410448,v1:127.0.0.1:6805/405410448] exists,up 2a90c41a-d91e-4c01-b1b8-9af4c2d95977 2024-04-25T07:11:49.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:49.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:49.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:49.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2024-04-25T07:11:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:11:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:49.267 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:11:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:49.709 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2231749179,v1:127.0.0.1:6819/2231749179] [v2:127.0.0.1:6820/2231749179,v1:127.0.0.1:6821/2231749179] exists,up f198c0c1-66e3-4ae7-9073-15d722aaf43f 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:49.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:49.712 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:11:49.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:49.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:11:50.140 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 38 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/513351488,v1:127.0.0.1:6811/513351488] [v2:127.0.0.1:6812/513351488,v1:127.0.0.1:6813/513351488] exists,up 993b915e-c902-451f-aaba-3eb24893ac73 2024-04-25T07:11:50.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:50.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:50.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:50.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:50.539 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:50.539 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:11:02.789422+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:17:21.139669+0000 0 0 2024-04-25T07:11:50.539 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:11:50.539 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:50.540 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:11:50.556 INFO:tasks.workunit.client.0.smithi134.stdout:wait for peering 2024-04-25T07:11:50.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2024-04-25T07:11:50.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:50.957 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:50.957 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:11:02.789422+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:17:21.139669+0000 0 0 2024-04-25T07:11:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:11:50.957 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:50.957 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:11:50.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2024-04-25T07:11:51.124 INFO:tasks.workunit.client.0.smithi134.stdout:killing divergent 1 2024-04-25T07:11:51.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2024-04-25T07:11:51.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:51.504 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:51.504 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:11:02.789422+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:17:21.139669+0000 0 0 2024-04-25T07:11:51.504 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:11:51.504 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:51.505 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:11:51.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:11:51.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:11:51.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:11:51.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:11:51.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:11:51.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:11:51.627 INFO:tasks.workunit.client.0.smithi134.stdout:reviving divergent 1 2024-04-25T07:11:51.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:11:51.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2024-04-25T07:11:51.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2024-04-25T07:11:52.023 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:11:52.023 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:11:02.789422+0000 41'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:17:21.139669+0000 0 0 2024-04-25T07:11:52.023 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:11:52.023 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:11:52.024 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:11:52.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:11:52.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:11:52.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:11:52.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:11:52.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:11:52.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:11:52.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:11:52.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:11:52.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:11:52.042 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:11:52.042 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:11:52.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:11:52.044 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:11:52.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:11:52.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:11:52.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:11:52.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:11:52.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:11:52.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:11:52.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:11:52.082 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:52.085+0000 7fb695a207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:52.085 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:52.085+0000 7fb695a207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:52.086 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:52.089+0000 7fb695a207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:11:52.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:11:52.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:11:52.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:11:52.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:11:52.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:11:52.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:11:52.447 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:11:52.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:11:52.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:11:52.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:11:52.620 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:52.621+0000 7fb695a207c0 -1 Falling back to public interface 2024-04-25T07:11:52.885 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/2231749179,v1:127.0.0.1:6819/2231749179] [v2:127.0.0.1:6820/2231749179,v1:127.0.0.1:6821/2231749179] exists,up f198c0c1-66e3-4ae7-9073-15d722aaf43f 2024-04-25T07:11:52.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:11:52.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:11:52.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:11:52.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2024-04-25T07:11:55.698 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:11:55.701+0000 7fb695a207c0 -1 osd.1 48 log_to_monitors true 2024-04-25T07:12:12.886 INFO:tasks.workunit.client.0.smithi134.stdout:allowing recovery 2024-04-25T07:12:12.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2024-04-25T07:12:12.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2024-04-25T07:12:13.292 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:12:13.292 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-04-25T07:11:57.667783+0000 48'102 53:156 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:43:53.274243+0000 0 0 2024-04-25T07:12:13.292 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:12:13.292 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:12:13.293 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:12:13.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T07:12:13.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:12:13.463 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:12:13.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T07:12:13.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:12:13.618 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:12:13.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2024-04-25T07:12:13.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:12:13.769 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:12:13.782 INFO:tasks.workunit.client.0.smithi134.stdout:reading divergent objects 2024-04-25T07:12:13.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2024-04-25T07:12:13.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2024-04-25T07:12:14.693 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:12:14.693 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 0 0 0 62667 0 0 18 84 18 active+clean 2024-04-25T07:11:57.667783+0000 48'102 53:156 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:09:59.441822+0000 0'0 2024-04-25T07:09:59.441822+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:43:53.274243+0000 0 0 2024-04-25T07:12:14.693 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:12:14.693 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:12:14.693 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:12:14.710 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2024-04-25T07:12:14.711 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2024-04-25T07:12:14.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:12:14.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:12:14.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:12:14.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:12:14.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:12:14.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:12:14.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:12:14.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:12:14.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2024-04-25T07:12:14.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-04-25T07:12:14.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:12:14.868 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:50.241+0000 7f53a23f5640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4628.0:1 2024-04-25T07:10:35.926215+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:12:14.868 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:50.241+0000 7f53a23f5640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4628.0:1 2024-04-25T07:10:35.926215+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:12:14.868 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:50.241+0000 7f53a23f5640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:12:14.868 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:50.241+0000 7f53a23f5640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2024-04-25T07:12:14.868 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:50.241+0000 7f53a23f5640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-04-25T07:12:14.868 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:50.241+0000 7f53a23f5640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:12:14.868 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:50.241+0000 7f53a23f5640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2024-04-25T07:12:14.868 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:50.241+0000 7f53a23f5640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4628.0:1 2024-04-25T07:10:35.926215+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:12:14.868 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:11:50.241+0000 7f53a23f5640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-04-25T07:12:14.869 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2024-04-25T07:12:14.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:12:14.873 INFO:tasks.workunit.client.0.smithi134.stdout:success 2024-04-25T07:12:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2024-04-25T07:12:14.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2024-04-25T07:12:14.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:12:14.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:12:15.411 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T07:12:15.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2024-04-25T07:12:15.428 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:12:15.428 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:12:15.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:12:15.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:12:15.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:12:20.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:12:20.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:12:20.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:12:20.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:12:20.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:12:20.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:12:20.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:12:20.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:12:20.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:12:20.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:12:20.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:12:20.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:12:20.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:12:20.756 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:12:20.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:12:20.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:12:20.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:12:20.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:20.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:12:20.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:12:20.759 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:20.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:12:20.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:12:20.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:12:20.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:12:20.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:20.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:12:20.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26105 2024-04-25T07:12:20.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:12:20.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:12:20.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:12:20.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:12:20.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:12:20.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:12:20.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:12:20.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:12:20.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:12:20.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:12:20.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:12:20.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:12:20.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:12:20.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:12:20.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:12:20.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:12:20.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:12:20.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:12:20.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:12:20.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:12:20.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:12:20.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:20.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:12:20.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:12:20.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:12:20.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:12:20.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:12:20.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:12:20.794 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:12:20.794 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:20.794 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:12:20.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26105 2024-04-25T07:12:20.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:12:20.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:12:20.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:12:20.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:12:20.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:20.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:12:20.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26105 2024-04-25T07:12:20.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:12:20.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:12:20.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:12:20.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:12:20.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2024-04-25T07:12:20.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2024-04-25T07:12:20.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2024-04-25T07:12:20.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2024-04-25T07:12:20.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2024-04-25T07:12:20.799 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2024-04-25T07:12:20.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2024-04-25T07:12:20.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2024-04-25T07:12:20.801 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:12:20.801 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:12:20.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2024-04-25T07:12:20.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:12:20.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:12:20.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:12:20.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:12:20.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:12:20.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:12:21.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:12:21.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:21.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:21.286 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:21.286 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:21.286 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:12:21.287 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:12:21.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:12:21.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:12:21.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:12:21.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:12:21.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:12:21.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:12:21.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:12:21.507 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:12:21.507 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:12:21.507 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:12:21.507 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:12:21.508 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:21.508 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:12:21.508 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-mon.a.asok 2024-04-25T07:12:21.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:12:21.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26105/ceph-mon.a.asok config get fsid 2024-04-25T07:12:21.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:12:21.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:12:21.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:12:21.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:12:21.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:12:21.626 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:12:21.626 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:12:21.626 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:12:21.627 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:12:21.627 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:21.627 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:12:21.628 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-mon.a.asok 2024-04-25T07:12:21.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:12:21.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26105/ceph-mon.a.asok config get mon_host 2024-04-25T07:12:21.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2024-04-25T07:12:21.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:12:21.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:12:21.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:12:21.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:12:21.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:12:21.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:12:22.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:12:22.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:22.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:22.678 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:22.678 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:22.678 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:12:22.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:12:22.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:12:22.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:12:22.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T07:12:22.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2024-04-25T07:12:22.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:12:22.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:12:22.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:22.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:22.781 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:22.781 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:22.781 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:12:22.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:12:22.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:12:22.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:22.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:12:22.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c3500745-9c36-472c-b5b0-17cf5b183183 2024-04-25T07:12:22.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c3500745-9c36-472c-b5b0-17cf5b183183' 2024-04-25T07:12:22.784 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 c3500745-9c36-472c-b5b0-17cf5b183183 2024-04-25T07:12:22.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:12:22.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBWAipmfSpgMBAAvqLkYsy19TEejnV+EQHzFQ== 2024-04-25T07:12:22.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBWAipmfSpgMBAAvqLkYsy19TEejnV+EQHzFQ=="}' 2024-04-25T07:12:22.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c3500745-9c36-472c-b5b0-17cf5b183183 -i td/divergent-priors/0/new.json 2024-04-25T07:12:23.134 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:12:23.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:12:23.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBWAipmfSpgMBAAvqLkYsy19TEejnV+EQHzFQ== --osd-uuid c3500745-9c36-472c-b5b0-17cf5b183183 2024-04-25T07:12:23.193 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:23.197+0000 7f979a7047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:23.211 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:23.213+0000 7f979a7047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:23.212 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:23.213+0000 7f979a7047c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:23.213 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:23.213+0000 7f979a7047c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:12:23.213 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:23.213+0000 7f979a7047c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:12:23.213 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:23.213+0000 7f979a7047c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:12:23.250 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:23.253+0000 7f979a7047c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:12:29.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:12:29.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:12:29.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:12:29.424 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:12:29.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:12:30.000 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:12:30.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:12:30.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:12:30.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:12:30.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:12:30.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:12:30.040 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:30.041+0000 7f42ba8d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:30.049 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:30.049+0000 7f42ba8d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:30.050 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:30.053+0000 7f42ba8d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:30.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:12:30.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:12:30.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:12:30.463 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:12:30.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:12:30.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:12:30.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:30.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:12:30.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:30.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:30.631 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:30.633+0000 7f42ba8d37c0 -1 Falling back to public interface 2024-04-25T07:12:30.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:31.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:31.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:31.890 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:12:31.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:12:31.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:31.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:32.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:33.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:33.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:33.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:12:33.331 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:12:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:33.339 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:33.341+0000 7f42ba8d37c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:12:33.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:34.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:34.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:34.954 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:12:34.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:12:34.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:34.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:35.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:36.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:36.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:36.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:12:36.455 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:12:36.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:36.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:36.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:37.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:37.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:37.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:12:37.903 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:12:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:38.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:39.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:39.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:39.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:12:39.459 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:12:39.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:39.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:39.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:40.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:40.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:40.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:12:40.914 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:12:40.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:40.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:41.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:42.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:42.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:42.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:12:42.344 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:12:42.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:42.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:42.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:43.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:43.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:43.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:12:43.775 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:12:43.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:43.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:44.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:45.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:45.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:45.258 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:12:45.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:12:45.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:45.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:45.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:46.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:46.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:46.686 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:12:46.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:12:46.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:46.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:47.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:48.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:48.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:48.737 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:12:48.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:12:48.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:48.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:49.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:50.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:50.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:50.624 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:12:50.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:12:50.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:50.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:51.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:52.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:52.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:52.464 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:12:52.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:12:52.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:52.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:53.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:54.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:54.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:54.140 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:12:54.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:12:54.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:54.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:55.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:55.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:55.670 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:12:55.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:12:55.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:55.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:56.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:12:57.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:12:57.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:12:57.559 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:12:57.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:12:57.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:12:57.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:12:57.998 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1293970793,v1:127.0.0.1:6803/1293970793] [v2:127.0.0.1:6804/1293970793,v1:127.0.0.1:6805/1293970793] exists,up c3500745-9c36-472c-b5b0-17cf5b183183 2024-04-25T07:12:58.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:12:58.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:12:58.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:12:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:12:58.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:12:58.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:12:58.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:12:58.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:12:58.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:12:58.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:12:58.002 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:12:58.002 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:12:58.002 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:12:58.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:12:58.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:12:58.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:12:58.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89b7ddae-ddc4-4e63-b12e-53901c5a0d0d 2024-04-25T07:12:58.005 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 89b7ddae-ddc4-4e63-b12e-53901c5a0d0d 2024-04-25T07:12:58.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 89b7ddae-ddc4-4e63-b12e-53901c5a0d0d' 2024-04-25T07:12:58.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:12:58.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB6AipmunNZAhAAuwnhmM/jEnrR0UKlTsxJ8A== 2024-04-25T07:12:58.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB6AipmunNZAhAAuwnhmM/jEnrR0UKlTsxJ8A=="}' 2024-04-25T07:12:58.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89b7ddae-ddc4-4e63-b12e-53901c5a0d0d -i td/divergent-priors/1/new.json 2024-04-25T07:12:58.617 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:12:58.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:12:58.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6AipmunNZAhAAuwnhmM/jEnrR0UKlTsxJ8A== --osd-uuid 89b7ddae-ddc4-4e63-b12e-53901c5a0d0d 2024-04-25T07:12:58.679 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:58.680+0000 7f74d5e147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:58.712 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:58.712+0000 7f74d5e147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:58.713 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:58.716+0000 7f74d5e147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:12:58.714 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:58.716+0000 7f74d5e147c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:12:58.714 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:58.716+0000 7f74d5e147c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:12:58.714 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:58.716+0000 7f74d5e147c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:12:58.758 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:12:58.760+0000 7f74d5e147c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:13:05.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:13:05.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:13:05.456 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:13:05.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:13:05.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:13:06.071 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:13:06.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:13:06.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:13:06.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:13:06.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:13:06.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:13:06.111 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:06.112+0000 7fe5a67e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:06.133 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:06.132+0000 7fe5a67e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:06.134 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:06.136+0000 7fe5a67e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:06.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:13:06.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:13:06.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:13:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:13:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:13:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:13:06.494 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:13:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:06.674 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:06.676+0000 7fe5a67e27c0 -1 Falling back to public interface 2024-04-25T07:13:06.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:07.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:07.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:07.934 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:13:07.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:13:07.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:07.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:08.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:09.165 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:09.164+0000 7fe5a67e27c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:13:09.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:09.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:09.359 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:13:09.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:13:09.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:09.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:09.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:10.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:10.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:10.992 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:13:10.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:13:10.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:10.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:11.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:12.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:12.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:12.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:13:12.590 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:13:12.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:12.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:13.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:14.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:14.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:14.035 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:13:14.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:13:14.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:14.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:14.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:15.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:15.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:15.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:13:15.522 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:13:15.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:15.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:16.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:17.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:17.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:17.150 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:13:17.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:13:17.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:17.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:17.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:18.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:18.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:18.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:13:18.606 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:13:18.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:18.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:19.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:20.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:20.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:20.029 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:13:20.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:13:20.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:20.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:20.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:21.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:21.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:21.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:13:21.562 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:13:21.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:21.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:22.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:23.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:23.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:23.148 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:13:23.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:13:23.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:23.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:24.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:25.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:25.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:25.177 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:13:25.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:13:25.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:25.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:26.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:27.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:27.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:27.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:13:27.592 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:13:27.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:27.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:28.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:29.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:29.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:29.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:13:29.421 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:13:29.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:29.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:30.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:31.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:31.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:31.332 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:13:31.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:13:31.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:31.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:32.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:33.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:33.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:33.186 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:13:33.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:13:33.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:33.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:33.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:34.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:34.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:34.806 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:13:34.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:13:34.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:34.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:13:35.308 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2462515974,v1:127.0.0.1:6811/2462515974] [v2:127.0.0.1:6812/2462515974,v1:127.0.0.1:6813/2462515974] exists,up 89b7ddae-ddc4-4e63-b12e-53901c5a0d0d 2024-04-25T07:13:35.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:13:35.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:13:35.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:13:35.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:13:35.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:13:35.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:13:35.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:13:35.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:13:35.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:13:35.313 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:13:35.313 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:13:35.313 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:13:35.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:13:35.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:13:35.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:13:35.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:13:35.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:13:35.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:13:35.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:13:35.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:13:35.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:13:35.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:13:35.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:13:35.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:13:35.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:13:35.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:13:35.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:13:35.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=00272bde-e741-4105-bc47-4625a8a19b02 2024-04-25T07:13:35.317 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 00272bde-e741-4105-bc47-4625a8a19b02 2024-04-25T07:13:35.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 00272bde-e741-4105-bc47-4625a8a19b02' 2024-04-25T07:13:35.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:13:35.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCfAipm3dzNFBAAaOjtTteatI79O8pX96KA0g== 2024-04-25T07:13:35.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCfAipm3dzNFBAAaOjtTteatI79O8pX96KA0g=="}' 2024-04-25T07:13:35.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 00272bde-e741-4105-bc47-4625a8a19b02 -i td/divergent-priors/2/new.json 2024-04-25T07:13:35.899 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:13:35.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:13:35.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCfAipm3dzNFBAAaOjtTteatI79O8pX96KA0g== --osd-uuid 00272bde-e741-4105-bc47-4625a8a19b02 2024-04-25T07:13:35.955 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:35.956+0000 7f003a3b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:35.957 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:35.960+0000 7f003a3b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:35.959 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:35.960+0000 7f003a3b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:35.959 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:35.960+0000 7f003a3b57c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:13:35.960 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:35.960+0000 7f003a3b57c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:13:35.960 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:35.960+0000 7f003a3b57c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:13:36.000 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:36.000+0000 7f003a3b57c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:13:42.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:13:42.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:13:42.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:13:42.418 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:13:42.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:13:42.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:13:42.990 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:13:42.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:13:42.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:13:42.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:13:42.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:13:43.031 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:43.032+0000 7f7487ff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:43.048 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:43.052+0000 7f7487ff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:43.050 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:43.052+0000 7f7487ff57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:13:43.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:13:43.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:13:43.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:13:43.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:13:43.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:13:43.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:43.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:13:43.415 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:13:43.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:43.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:43.612+0000 7f7487ff57c0 -1 Falling back to public interface 2024-04-25T07:13:43.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:44.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:44.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:44.848 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:13:44.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:13:44.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:44.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:45.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:46.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:46.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:46.296 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:13:46.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:13:46.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:46.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:46.299 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:13:46.299+0000 7f7487ff57c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:13:47.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:48.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:48.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:48.056 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:13:48.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:13:48.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:48.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:48.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:49.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:49.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:49.585 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:13:49.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:13:49.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:49.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:50.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:51.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:51.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:51.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:13:51.155 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:13:51.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:51.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:51.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:52.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:52.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:52.614 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:13:52.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:13:52.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:52.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:53.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:54.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:54.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:54.195 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:13:54.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:13:54.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:54.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:54.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:55.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:55.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:55.681 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:13:55.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:13:55.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:55.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:56.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:57.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:57.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:57.257 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:13:57.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:13:57.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:57.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:57.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:13:58.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:13:58.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:13:58.696 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:13:58.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:13:58.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:13:58.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:13:59.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:00.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:00.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:00.587 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:14:00.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:14:00.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:00.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:01.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:02.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:02.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:02.458 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:14:02.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:14:02.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:02.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:03.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:04.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:04.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:04.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:14:04.344 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:14:04.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:04.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:05.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:06.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:06.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:06.113 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:14:06.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:14:06.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:06.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:07.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:08.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:08.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:08.444 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:14:08.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:14:08.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:08.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:09.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:14:10.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:14:10.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:14:10.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:14:10.646 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:14:10.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:14:10.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:14:11.086 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2003862894,v1:127.0.0.1:6819/2003862894] [v2:127.0.0.1:6820/2003862894,v1:127.0.0.1:6821/2003862894] exists,up 00272bde-e741-4105-bc47-4625a8a19b02 2024-04-25T07:14:11.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:14:11.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:14:11.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:14:11.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2024-04-25T07:14:11.904 INFO:tasks.workunit.client.0.smithi134.stderr:noout is set 2024-04-25T07:14:11.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2024-04-25T07:14:12.850 INFO:tasks.workunit.client.0.smithi134.stderr:noin is set 2024-04-25T07:14:12.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2024-04-25T07:14:13.796 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T07:14:13.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2024-04-25T07:14:13.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:14:14.341 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T07:14:14.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:14:15.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2024-04-25T07:14:16.098 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 3 2024-04-25T07:14:16.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2024-04-25T07:14:16.902 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 min_size to 2 2024-04-25T07:14:16.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2024-04-25T07:14:16.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:14:16.925 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:14:17.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:14:17.430 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:14:17.430 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:14:17.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:14:17.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:17.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:14:17.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-04-25T07:14:17.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-04-25T07:14:17.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-04-25T07:14:17.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:17.598 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:14:17.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672974 2024-04-25T07:14:17.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672974 2024-04-25T07:14:17.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672974' 2024-04-25T07:14:17.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:17.794 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:14:18.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509447 2024-04-25T07:14:18.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509447 2024-04-25T07:14:18.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672974 2-64424509447' 2024-04-25T07:14:18.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:18.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-04-25T07:14:18.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:18.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:14:18.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-04-25T07:14:18.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:18.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-04-25T07:14:18.010 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836502 2024-04-25T07:14:18.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-25T07:14:18.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:14:18.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836502 2024-04-25T07:14:18.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:14:19.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:14:19.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:14:19.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836502 2024-04-25T07:14:19.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:19.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672974 2024-04-25T07:14:19.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:19.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:14:19.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672974 2024-04-25T07:14:19.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:19.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672974 2024-04-25T07:14:19.967 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672974 2024-04-25T07:14:19.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2024-04-25T07:14:19.967 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:14:20.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672974 -lt 42949672974 2024-04-25T07:14:20.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:20.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509447 2024-04-25T07:14:20.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:20.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:14:20.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509447 2024-04-25T07:14:20.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:20.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509447 2024-04-25T07:14:20.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509447' 2024-04-25T07:14:20.387 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509447 2024-04-25T07:14:20.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:14:20.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509447 -lt 64424509447 2024-04-25T07:14:20.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2024-04-25T07:14:20.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:14:20.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:14:20.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:14:20.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:14:20.833 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:14:20.834 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:14:20.834 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:14:20.834 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:14:20.834 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:14:20.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:14:20.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:14:20.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:14:20.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:14:20.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:14:20.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:14:21.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:14:21.356 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:14:21.356 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:14:21.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:14:21.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:21.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:14:21.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836505 2024-04-25T07:14:21.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836505 2024-04-25T07:14:21.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505' 2024-04-25T07:14:21.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:21.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:14:21.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-04-25T07:14:21.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-04-25T07:14:21.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672977' 2024-04-25T07:14:21.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:21.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:14:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-04-25T07:14:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-04-25T07:14:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836505 1-42949672977 2-64424509450' 2024-04-25T07:14:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:21.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836505 2024-04-25T07:14:21.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:21.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:14:21.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836505 2024-04-25T07:14:21.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836505 2024-04-25T07:14:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-04-25T07:14:21.841 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836505 2024-04-25T07:14:21.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:14:22.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836505 2024-04-25T07:14:22.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:14:23.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:14:23.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:14:23.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836505 2024-04-25T07:14:23.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:23.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:23.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-04-25T07:14:23.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:14:23.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-04-25T07:14:23.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:23.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-04-25T07:14:23.741 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672977 2024-04-25T07:14:23.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-25T07:14:23.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:14:24.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672977 -lt 42949672977 2024-04-25T07:14:24.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:24.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-04-25T07:14:24.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:24.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:14:24.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:24.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-04-25T07:14:24.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-04-25T07:14:24.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-04-25T07:14:24.193 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509450 2024-04-25T07:14:24.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:14:24.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509450 2024-04-25T07:14:24.615 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:14:24.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:24.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:25.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:14:25.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:25.186 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:25.186 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:25.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:25.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:25.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:25.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:25.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:14:25.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:25.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:25.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:26.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:14:26.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:14:26.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:14:26.161 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2024-04-25T07:14:26.162 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2024-04-25T07:14:26.562 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:14:26.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2024-04-25T07:14:26.578 INFO:tasks.workunit.client.0.smithi134.stdout:primary and soon to be divergent is 1 2024-04-25T07:14:26.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2024-04-25T07:14:26.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:14:26.957 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:14:26.958 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:14:15.574341+0000 0'0 28:18 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:39:24.299570+0000 0 0 2024-04-25T07:14:26.958 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:14:26.958 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:14:26.958 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:14:26.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2024-04-25T07:14:26.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T07:14:26.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2024-04-25T07:14:26.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2024-04-25T07:14:26.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T07:14:26.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2024-04-25T07:14:26.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2024-04-25T07:14:26.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2024-04-25T07:14:26.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2024-04-25T07:14:26.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2024-04-25T07:14:26.974 INFO:tasks.workunit.client.0.smithi134.stdout:writing initial objects 2024-04-25T07:14:26.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2024-04-25T07:14:26.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2024-04-25T07:14:26.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:26.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2024-04-25T07:14:27.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:27.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2024-04-25T07:14:27.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:27.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2024-04-25T07:14:27.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:27.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2024-04-25T07:14:27.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:27.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2024-04-25T07:14:27.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:27.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2024-04-25T07:14:28.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:28.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2024-04-25T07:14:28.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:28.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2024-04-25T07:14:28.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:28.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2024-04-25T07:14:28.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:28.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2024-04-25T07:14:28.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:28.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2024-04-25T07:14:28.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:28.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2024-04-25T07:14:29.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:29.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2024-04-25T07:14:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2024-04-25T07:14:29.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:29.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2024-04-25T07:14:29.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:29.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2024-04-25T07:14:29.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:29.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2024-04-25T07:14:30.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:30.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2024-04-25T07:14:30.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:30.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2024-04-25T07:14:30.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:30.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2024-04-25T07:14:30.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:30.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2024-04-25T07:14:30.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:30.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2024-04-25T07:14:31.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:31.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2024-04-25T07:14:31.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:31.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2024-04-25T07:14:31.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:31.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2024-04-25T07:14:31.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:31.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2024-04-25T07:14:31.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:31.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2024-04-25T07:14:31.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:31.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2024-04-25T07:14:32.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:32.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2024-04-25T07:14:32.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:32.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2024-04-25T07:14:32.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:32.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2024-04-25T07:14:32.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:32.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2024-04-25T07:14:32.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:32.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2024-04-25T07:14:33.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:33.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2024-04-25T07:14:33.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:33.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2024-04-25T07:14:33.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:33.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2024-04-25T07:14:33.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:33.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2024-04-25T07:14:33.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:33.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2024-04-25T07:14:33.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:33.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2024-04-25T07:14:34.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:34.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2024-04-25T07:14:34.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:34.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2024-04-25T07:14:34.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:34.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2024-04-25T07:14:34.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:34.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2024-04-25T07:14:34.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:34.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2024-04-25T07:14:35.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:35.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2024-04-25T07:14:35.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:35.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2024-04-25T07:14:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2024-04-25T07:14:35.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:35.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2024-04-25T07:14:35.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:35.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2024-04-25T07:14:36.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:36.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2024-04-25T07:14:36.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:36.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2024-04-25T07:14:36.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:36.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2024-04-25T07:14:36.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:36.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2024-04-25T07:14:36.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:36.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2024-04-25T07:14:37.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:37.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2024-04-25T07:14:37.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:37.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2024-04-25T07:14:37.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:37.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2024-04-25T07:14:37.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:37.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2024-04-25T07:14:37.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:37.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2024-04-25T07:14:37.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:37.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2024-04-25T07:14:38.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:38.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2024-04-25T07:14:38.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:38.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2024-04-25T07:14:38.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:38.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2024-04-25T07:14:38.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:38.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2024-04-25T07:14:38.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:38.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2024-04-25T07:14:39.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:39.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2024-04-25T07:14:39.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:39.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2024-04-25T07:14:39.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:39.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2024-04-25T07:14:39.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:39.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2024-04-25T07:14:39.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:39.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2024-04-25T07:14:40.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:40.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2024-04-25T07:14:40.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:40.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2024-04-25T07:14:40.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:40.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2024-04-25T07:14:40.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:40.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2024-04-25T07:14:40.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:40.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2024-04-25T07:14:40.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:40.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2024-04-25T07:14:41.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:41.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2024-04-25T07:14:41.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:41.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2024-04-25T07:14:41.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:41.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2024-04-25T07:14:41.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:41.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2024-04-25T07:14:41.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:41.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2024-04-25T07:14:42.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:42.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2024-04-25T07:14:42.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:42.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2024-04-25T07:14:42.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:42.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2024-04-25T07:14:42.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:42.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2024-04-25T07:14:42.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:42.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2024-04-25T07:14:42.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:42.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2024-04-25T07:14:43.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:43.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2024-04-25T07:14:43.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:43.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2024-04-25T07:14:43.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:43.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2024-04-25T07:14:43.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:43.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2024-04-25T07:14:43.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:43.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2024-04-25T07:14:44.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:44.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2024-04-25T07:14:44.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:44.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2024-04-25T07:14:44.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:44.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2024-04-25T07:14:44.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:44.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2024-04-25T07:14:44.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:44.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2024-04-25T07:14:44.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:44.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2024-04-25T07:14:45.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:45.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2024-04-25T07:14:45.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2024-04-25T07:14:45.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2024-04-25T07:14:45.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:14:45.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2024-04-25T07:14:45.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:14:45.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:14:45.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:14:45.468 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:14:45.469 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:14:45.469 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:14:45.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:14:45.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:14:45.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:14:45.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:14:45.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:14:45.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:14:45.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:14:45.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:14:45.537 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:14:45.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:14:45.966 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:14:45.966 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:14:45.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:14:45.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:45.966 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:14:46.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836512 2024-04-25T07:14:46.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836512 2024-04-25T07:14:46.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512' 2024-04-25T07:14:46.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:46.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:14:46.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672984 2024-04-25T07:14:46.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672984 2024-04-25T07:14:46.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672984' 2024-04-25T07:14:46.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:14:46.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-04-25T07:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-04-25T07:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672984 2-64424509457' 2024-04-25T07:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:46.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836512 2024-04-25T07:14:46.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:46.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:14:46.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836512 2024-04-25T07:14:46.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:46.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836512 2024-04-25T07:14:46.501 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836512 2024-04-25T07:14:46.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-04-25T07:14:46.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:14:46.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836512 2024-04-25T07:14:46.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:14:47.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:14:47.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:14:48.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836512 -lt 21474836512 2024-04-25T07:14:48.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:48.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672984 2024-04-25T07:14:48.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:48.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:14:48.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672984 2024-04-25T07:14:48.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:48.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672984 2024-04-25T07:14:48.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-04-25T07:14:48.370 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672984 2024-04-25T07:14:48.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:14:48.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672984 -lt 42949672984 2024-04-25T07:14:48.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:14:48.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-04-25T07:14:48.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:14:48.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:14:48.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-04-25T07:14:48.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:14:48.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-04-25T07:14:48.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-04-25T07:14:48.802 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509457 2024-04-25T07:14:48.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:14:49.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509457 2024-04-25T07:14:49.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:14:49.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:49.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:14:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:14:49.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:14:49.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:14:49.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:14:49.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:14:49.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:14:49.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:14:50.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:14:50.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:14:50.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:14:50.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:14:50.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:14:50.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:14:50.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:14:50.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2024-04-25T07:14:50.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:14:50.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:14:50.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:14:50.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:14:51.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2024-04-25T07:14:51.203 INFO:tasks.workunit.client.0.smithi134.stdout:blackholing osds 0 2 2024-04-25T07:14:51.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2024-04-25T07:14:51.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:14:51.678 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:14:51.679 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:14:15.574341+0000 28'100 28:118 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:39:24.299570+0000 0 0 2024-04-25T07:14:51.679 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:14:51.679 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:14:51.679 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:14:51.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:51.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2024-04-25T07:14:51.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:14:51.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:14:51.697 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:14:51.697 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:51.697 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:14:51.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.0.asok 2024-04-25T07:14:51.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:14:51.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:14:51.802 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:14:51.802 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:14:51.802 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:14:51.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:14:51.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2024-04-25T07:14:51.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:14:51.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:14:51.816 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:14:51.816 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:14:51.817 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:14:51.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.2.asok 2024-04-25T07:14:51.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:14:51.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:14:51.913 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:14:51.914 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:14:51.914 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:14:51.925 INFO:tasks.workunit.client.0.smithi134.stdout:create new divergent objects 2024-04-25T07:14:51.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2024-04-25T07:14:51.925 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2024-04-25T07:14:51.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:14:51.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:14:51.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2024-04-25T07:14:51.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2024-04-25T07:14:51.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2024-04-25T07:14:51.928 INFO:tasks.workunit.client.0.smithi134.stdout:writing divergent objects 2024-04-25T07:14:51.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2024-04-25T07:14:51.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:14:51.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:14:51.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2024-04-25T07:14:51.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2024-04-25T07:14:51.931 INFO:tasks.workunit.client.0.smithi134.stdout:remove divergent objects 2024-04-25T07:14:51.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2024-04-25T07:14:51.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2024-04-25T07:14:51.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:14:51.933 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2024-04-25T07:14:51.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2024-04-25T07:14:51.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:14:51.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2024-04-25T07:14:51.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2024-04-25T07:14:51.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2024-04-25T07:14:51.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2024-04-25T07:14:51.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2024-04-25T07:15:01.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2024-04-25T07:15:01.944 INFO:tasks.workunit.client.0.smithi134.stderr:rados: no process found 2024-04-25T07:15:01.944 INFO:tasks.workunit.client.0.smithi134.stdout:killing all the osds 2024-04-25T07:15:01.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2024-04-25T07:15:01.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:15:02.357 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:15:02.358 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 active+clean 2024-04-25T07:14:15.574341+0000 28'106 28:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:39:24.299570+0000 0 0 2024-04-25T07:15:02.358 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:15:02.358 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:15:02.358 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:15:02.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:15:02.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:15:02.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:15:02.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:15:02.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:15:02.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:15:02.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:15:02.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T07:15:02.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2024-04-25T07:15:03.010 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:15:03.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T07:15:03.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2024-04-25T07:15:03.652 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already down. 2024-04-25T07:15:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2024-04-25T07:15:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2024-04-25T07:15:04.355 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already down. 2024-04-25T07:15:04.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:15:04.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2024-04-25T07:15:04.890 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T07:15:04.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:15:04.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2024-04-25T07:15:05.480 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already out. 2024-04-25T07:15:05.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2024-04-25T07:15:05.498 INFO:tasks.workunit.client.0.smithi134.stdout:bringing up non_divergent 0 2 2024-04-25T07:15:05.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:15:05.878 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:15:05.879 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 0 0 0 61166 0 0 16 90 16 stale+active+clean 2024-04-25T07:14:15.574341+0000 28'106 28:124 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:39:24.299570+0000 0 0 2024-04-25T07:15:05.879 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:15:05.879 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:15:05.879 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:15:05.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:15:05.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2024-04-25T07:15:05.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:15:05.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:15:05.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:15:05.896 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:15:05.898 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:15:05.898 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:15:05.898 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:15:05.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:15:05.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:15:05.900 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:15:05.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:15:05.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:15:05.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:15:05.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:15:05.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:15:05.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:15:05.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:15:05.937 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:05.938+0000 7f73f7cbc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:05.966 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:05.966+0000 7f73f7cbc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:05.968 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:05.970+0000 7f73f7cbc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:06.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:15:06.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:15:06.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:15:06.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:15:06.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:15:06.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:06.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:15:06.309 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:15:06.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:06.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:15:06.513 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:06.514+0000 7f73f7cbc7c0 -1 Falling back to public interface 2024-04-25T07:15:06.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:07.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:07.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:07.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:15:07.747 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:15:07.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:07.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:15:08.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:09.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:09.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:09.183 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:15:09.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:15:09.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:09.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:15:09.524 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:09.526+0000 7f73f7cbc7c0 -1 osd.0 28 log_to_monitors true 2024-04-25T07:15:09.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:10.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:10.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:10.618 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:15:10.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:15:10.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:10.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:15:11.053 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/3564868565,v1:127.0.0.1:6803/3564868565] [v2:127.0.0.1:6804/3564868565,v1:127.0.0.1:6805/3564868565] exists,up c3500745-9c36-472c-b5b0-17cf5b183183 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:15:11.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:15:11.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:15:11.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:15:11.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:15:11.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:15:11.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:15:11.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:15:11.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:15:11.057 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:15:11.057 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:15:11.057 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:15:11.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:15:11.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:15:11.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:15:11.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:15:11.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:15:11.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:15:11.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:15:11.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:15:11.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:15:11.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:15:11.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:15:11.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:15:11.058 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:15:11.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:15:11.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:15:11.059 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:15:11.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:15:11.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:15:11.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:15:11.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:15:11.092 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:11.094+0000 7f884517a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:11.095 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:11.094+0000 7f884517a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:11.096 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:11.098+0000 7f884517a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:15:11.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:15:11.466 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:11.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:11.670 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:11.670+0000 7f884517a7c0 -1 Falling back to public interface 2024-04-25T07:15:11.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:12.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:12.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:12.875 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:15:12.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:15:12.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:12.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:13.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:14.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:14.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:14.317 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:15:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:15:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:14.703 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:15:14.702+0000 7f884517a7c0 -1 osd.2 28 log_to_monitors true 2024-04-25T07:15:14.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:15:15.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:15:15.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:15:15.768 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:15:15.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:15:15.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:15:15.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:15:16.207 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/3384236760,v1:127.0.0.1:6811/3384236760] [v2:127.0.0.1:6812/3384236760,v1:127.0.0.1:6813/3384236760] exists,up 00272bde-e741-4105-bc47-4625a8a19b02 2024-04-25T07:15:16.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:15:16.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:15:16.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:15:16.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:15:16.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2024-04-25T07:15:16.774 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already in. 2024-04-25T07:15:16.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:15:16.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2024-04-25T07:15:17.366 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already in. 2024-04-25T07:15:17.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:15:17.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2024-04-25T07:15:17.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:15:17.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:15:17.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:15:17.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:15:17.385 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:15:17.385 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:15:17.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:15:17.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:15:17.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:15:17.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:15:17.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:15:17.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:15:17.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:15:17.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:15:17.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:15:17.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:15:17.897 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:15:17.897 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:15:17.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:15:17.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:15:17.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:15:18.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855364 2024-04-25T07:15:18.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855364 2024-04-25T07:15:18.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855364' 2024-04-25T07:15:18.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:15:18.113 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:15:18.239 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:15:18.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:15:18.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:15:18.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:15:18.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:15:18.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:15:18.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789955 2024-04-25T07:15:18.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789955 2024-04-25T07:15:18.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855364 2-158913789955' 2024-04-25T07:15:18.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:15:18.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-150323855364 2024-04-25T07:15:18.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:15:18.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:15:18.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-150323855364 2024-04-25T07:15:18.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:15:18.405 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 150323855364 2024-04-25T07:15:18.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855364 2024-04-25T07:15:18.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 150323855364' 2024-04-25T07:15:18.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:15:18.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855362 -lt 150323855364 2024-04-25T07:15:18.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:15:19.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:15:19.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:15:20.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855364 -lt 150323855364 2024-04-25T07:15:20.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:15:20.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-158913789955 2024-04-25T07:15:20.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:15:20.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:15:20.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-158913789955 2024-04-25T07:15:20.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:15:20.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789955 2024-04-25T07:15:20.269 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 158913789955 2024-04-25T07:15:20.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 158913789955' 2024-04-25T07:15:20.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:15:20.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789955 -lt 158913789955 2024-04-25T07:15:20.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:15:20.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:20.683 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:21.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:15:21.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:21.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:21.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:21.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:21.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:21.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:21.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:21.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:21.685 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:21.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:21.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:22.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:22.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:15:22.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:15:22.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:15:22.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:22.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:15:22.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:22.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:22.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:22.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:22.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:22.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:22.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:22.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:22.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:22.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:22.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:22.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:23.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:23.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:23.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:23.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:23.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:23.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:23.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:23.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:23.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:23.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:23.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:23.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:15:23.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:23.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:15:24.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:24.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:24.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:24.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:24.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:24.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:24.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:24.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:24.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:24.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:24.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:24.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:25.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:25.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:25.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:25.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:25.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:25.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:25.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:25.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:25.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:25.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:25.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:25.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:15:25.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:25.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:15:26.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:26.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:26.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:26.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:26.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:26.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:26.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:26.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:26.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:26.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:26.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:26.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:26.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:26.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:27.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:27.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:27.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:15:27.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:27.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:15:28.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:28.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:28.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:28.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:28.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:28.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:28.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:28.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:28.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:28.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:28.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:28.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:29.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:29.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:29.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:29.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:29.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:29.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:29.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:29.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:29.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:15:29.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:29.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:15:31.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:31.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:31.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:31.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:31.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:31.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:31.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:31.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:31.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:31.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:31.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:31.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:32.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:32.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:32.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:32.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:32.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:32.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:32.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:32.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:32.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:33.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:33.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:33.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:15:33.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:33.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:15:36.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:36.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:36.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:36.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:36.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:36.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:36.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:36.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:36.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:36.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:36.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:36.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:37.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:37.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:37.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:37.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:37.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:37.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:37.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:37.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:37.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:37.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:37.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:37.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:15:37.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:37.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:15:44.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:44.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:44.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:44.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:44.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:44.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:44.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:44.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:44.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:44.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:44.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:44.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:45.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:45.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:45.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:45.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:45.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:45.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:45.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:45.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:45.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:45.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:45.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:45.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:15:45.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:15:45.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:15:53.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:15:53.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:53.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:53.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:53.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:53.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:53.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:53.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:15:53.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:15:53.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:15:53.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:53.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:54.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:15:54.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:15:54.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:15:54.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:15:54.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:15:54.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:15:54.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:15:54.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:15:54.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:15:54.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:15:54.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:15:54.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:15:54.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:15:55.274 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:15:55.274 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_fingerprint": "543d48d8-d6a1-4a5d-89cd-5ba9694a6bd9", 2024-04-25T07:15:55.274 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:55.274 INFO:tasks.workunit.client.0.smithi134.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:15:55.274 INFO:tasks.workunit.client.0.smithi134.stdout: "timestamp": "2024-04-25T07:15:55.277711+0000", 2024-04-25T07:15:55.274 INFO:tasks.workunit.client.0.smithi134.stdout: "tag": "", 2024-04-25T07:15:55.274 INFO:tasks.workunit.client.0.smithi134.stdout: "health": { 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "status": "HEALTH_WARN", 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "checks": { 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 17 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [], 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "OSD_DOWN": { 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 osds down", 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:15:55.275 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "osd.1 (root=default,host=smithi134) is down" 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: "PG_DEGRADED": { 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:15:55.276 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:15:55.277 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: "mutes": [] 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_first_committed": 1, 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_last_committed": 1, 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap": { 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:15:55.278 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:12:20.832324Z", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:12:20.832324Z", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release": 19, 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release_name": "squid", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "election_strategy": 1, 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "disallowed_leaders: ": "", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": false, 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "tiebreaker_mon": "", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_ranks: ": "", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "features": { 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "persistent": [ 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "kraken", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "luminous", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "mimic", 2024-04-25T07:15:55.279 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap-prune", 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "nautilus", 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "octopus", 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "pacific", 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "elector-pinging", 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "quincy", 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "reef", 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "squid" 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "optional": [] 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "mons": [ 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "rank": 0, 2024-04-25T07:15:55.280 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "a", 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 0 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 0, 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 0, 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_location": "{}" 2024-04-25T07:15:55.281 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "quorum": [ 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: 0 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap": { 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 40, 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:12:21.809794+0000", 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:15:17.693066+0000", 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "last_up_change": "2024-04-25T07:15:15.038151+0000", 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "last_in_change": "2024-04-25T07:15:17.193188+0000", 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:15:55.282 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_num": 5800384, 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_set": [ 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "nodown", 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "noin", 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "noout", 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "pglog_hardlimit", 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snapdirs", 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "recovery_deletes", 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "sortbitwise" 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_version": 9, 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_snapshot": "", 2024-04-25T07:15:55.283 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_max": 1, 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "max_osd": 3, 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "min_compat_client": "jewel", 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "require_osd_release": "squid", 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "allow_crimson": false, 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "pools": [ 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_name": "test", 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "create_time": "2024-04-25T07:14:14.126087+0000", 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": 1, 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_names": "hashpspool", 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:15:55.284 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 3, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "min_size": 2, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_rule": 0, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "object_hash": 2, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num": 1, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num": 1, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num_target": 1, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_target": 1, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_pending": 1, 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pg_merge_meta": { 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "source_pgid": "0.0", 2024-04-25T07:15:55.285 INFO:tasks.workunit.client.0.smithi134.stdout: "ready_epoch": 0, 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_started": 0, 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": 0, 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "source_version": "0'0", 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "target_version": "0'0" 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "last_change": "28", 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend": "0", 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "auid": 0, 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_seq": 0, 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_epoch": 0, 2024-04-25T07:15:55.286 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_snaps": [], 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps": "[]", 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_bytes": 0, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_objects": 0, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "tiers": [], 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "tier_of": -1, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "read_tier": -1, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "write_tier": -1, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_mode": "none", 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_bytes": 0, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_objects": 0, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_flush_age": 0, 2024-04-25T07:15:55.287 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_evict_age": 0, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profile": "", 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_params": { 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "none" 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_period": 0, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_count": 0, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "use_gmt_hitset": true, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "grade_table": [], 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "stripe_width": 0, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "expected_num_objects": 0, 2024-04-25T07:15:55.288 INFO:tasks.workunit.client.0.smithi134.stdout: "fast_read": false, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "options": {}, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "application_metadata": {}, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "read_balance": { 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "score_type": "Fair distribution", 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "score_acting": 2.9900000095367432, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "score_stable": 2.9900000095367432, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_acting": 2, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_stable": 2, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity": 1, 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.289 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "osds": [ 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "c3500745-9c36-472c-b5b0-17cf5b183183", 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 5, 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 28, 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 35, 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 39, 2024-04-25T07:15:55.290 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 29, 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.291 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.292 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.293 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.294 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6803/3564868565", 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6805/3564868565", 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3564868565", 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3564868565", 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.295 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "89b7ddae-ddc4-4e63-b12e-53901c5a0d0d", 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 0, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 10, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 23, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 30, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:15:55.296 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.297 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.298 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.299 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/2462515974", 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/2462515974", 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2462515974", 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2462515974", 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:15:55.300 INFO:tasks.workunit.client.0.smithi134.stdout: "exists" 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "00272bde-e741-4105-bc47-4625a8a19b02", 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 15, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 28, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 37, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 0, 2024-04-25T07:15:55.301 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 31, 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.302 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.303 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.304 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/3384236760", 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/3384236760", 2024-04-25T07:15:55.305 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3384236760", 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3384236760", 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_xinfo": [ 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:15:02.831126+0000", 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:15:55.306 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:12:56.907586+0000", 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:15:03.438039+0000", 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:13:33.522143+0000", 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:15:55.307 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:15:04.072862+0000", 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:14:10.149256+0000", 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap": [], 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_items": [], 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_primaries": [], 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_temp": [], 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_temp": [], 2024-04-25T07:15:55.308 INFO:tasks.workunit.client.0.smithi134.stdout: "blocklist": {}, 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "range_blocklist": {}, 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profiles": { 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "default": { 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "2", 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van" 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps_queue": [], 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "new_removed_snaps": [], 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "new_purged_snaps": [], 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_node_flags": {}, 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "device_class_flags": {}, 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": { 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_enabled": false, 2024-04-25T07:15:55.309 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_bucket_count": 0, 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_bucket": 0 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_metadata": [ 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6804/3564868565,v1:127.0.0.1:6805/3564868565]", 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:15:55.310 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:15:55.311 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:12:28.750502Z", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6802/3564868565,v1:127.0.0.1:6803/3564868565]", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3564868565,v1:127.0.0.1:6809/3564868565]", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3564868565,v1:127.0.0.1:6807/3564868565]", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:15:55.312 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/2462515974,v1:127.0.0.1:6813/2462515974]", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:15:55.313 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:55.314 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:13:04.783221Z", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/2462515974,v1:127.0.0.1:6811/2462515974]", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2462515974,v1:127.0.0.1:6817/2462515974]", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2462515974,v1:127.0.0.1:6815/2462515974]", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:15:55.315 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/3384236760,v1:127.0.0.1:6813/3384236760]", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:15:55.316 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:55.317 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:13:41.717511Z", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/3384236760,v1:127.0.0.1:6811/3384236760]", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3384236760,v1:127.0.0.1:6817/3384236760]", 2024-04-25T07:15:55.318 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3384236760,v1:127.0.0.1:6815/3384236760]", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.319 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_clean_epochs": { 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": { 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "per_pool": [] 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_epochs": [ 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 35 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 37 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.320 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_first_committed": 1, 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_last_committed": 40, 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "crushmap": { 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": [ 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.0", 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.1", 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.321 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.2", 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: "types": [ 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 0, 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd" 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "host" 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.322 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 2, 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "chassis" 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 3, 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "rack" 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 4, 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "row" 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 5, 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pdu" 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.323 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 6, 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pod" 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 7, 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "room" 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 8, 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "datacenter" 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 9, 2024-04-25T07:15:55.324 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "zone" 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 10, 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "region" 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "root" 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: "buckets": [ 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -1, 2024-04-25T07:15:55.325 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default", 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.326 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -2, 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default~hdd", 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:15:55.327 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134", 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:15:55.328 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.329 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134~hdd", 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:15:55.330 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: "rules": [ 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 0, 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "replicated_rule", 2024-04-25T07:15:55.331 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_firstn", 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:15:55.332 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "tunables": { 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_tries": 0, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_total_tries": 50, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_stable": 1, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_descents": 100, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_collision_tries": 100, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "straw_calc_version": 1, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "profile": "jewel", 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_tunables": 1, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "legacy_tunables": 0, 2024-04-25T07:15:55.333 INFO:tasks.workunit.client.0.smithi134.stdout: "minimum_required_version": "jewel", 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables": 1, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables2": 1, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v2_rules": 0, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables3": 1, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v3_rules": 0, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v4_buckets": 1, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables5": 1, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v5_rules": 0, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "has_msr_rules": 0 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_args": {} 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "fsmap": { 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "default_fscid": -1, 2024-04-25T07:15:55.334 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": { 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": {}, 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "ro_compat": {}, 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "incompat": { 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_1": "base v0.20", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_8": "no anchor table", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_9": "file layout v2", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_11": "minor log segments", 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.335 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_flags": { 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "enable_multiple": true, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "ever_enabled_multiple": true 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "standbys": [], 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "filesystems": [] 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "auth": { 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 5, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "num_secrets": 3 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_active": 1, 2024-04-25T07:15:55.336 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_unknown": 0, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osd": 3, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_sum": { 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 61700, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:15:55.337 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 100, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 100, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:15:55.338 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:15:55.339 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 0, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 0, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 16, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 16, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 2, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 2, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 0 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum": { 2024-04-25T07:15:55.340 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83596, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2080, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314489204, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036944896, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2129920, 2024-04-25T07:15:55.341 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 450462, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:15:55.342 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum_by_class": { 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "hdd": { 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:15:55.343 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83596, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2080, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314489204, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036944896, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2129920, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 450462, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:55.344 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:15:55.345 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_stats": [ 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 61700, 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:15:55.346 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 100, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 100, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:15:55.347 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:15:55.348 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 1220608, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 184566, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 16, 2024-04-25T07:15:55.349 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 16, 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 2, 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 2, 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 3 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_stats": [ 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 150323855371 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 42949672987 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:15:55.350 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 158913789962 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_state": [ 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "active+undersized+degraded", 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 1 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_osd": [ 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.351 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 1, 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 0, 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": [ 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:15:55.352 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": {} 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "servicemap": { 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 5, 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:15:07.310073+0000", 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "services": {} 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_last_committed": 95, 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_first_committed": 1, 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_last_committed": 378, 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "paxos": { 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 519, 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pn": 0, 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: "accepted_pn": 0 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:15:55.353 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:15:55.354 INFO:tasks.workunit.client.0.smithi134.stderr:report 2154229490 2024-04-25T07:15:55.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:15:55.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2024-04-25T07:15:55.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2024-04-25T07:15:55.355 INFO:tasks.workunit.client.0.smithi134.stdout:writing non-divergent object existing_4 2024-04-25T07:15:55.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2024-04-25T07:15:55.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:15:55.714 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:15:55.714 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:15:17.984932+0000 28'100 40:125 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:10:29.637545+0000 0 0 2024-04-25T07:15:55.714 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:15:55.714 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:15:55.714 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:15:55.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:15:55.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:15:55.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2024-04-25T07:15:55.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:15:55.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:15:55.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:15:55.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:15:55.859 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:15:55.859 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:15:55.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:15:55.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:15:55.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:15:55.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:15:55.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:15:55.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:15:55.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:15:55.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:15:55.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:15:56.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:15:56.351 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:15:56.352 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:15:56.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:15:56.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:15:56.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:15:56.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855374 2024-04-25T07:15:56.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855374 2024-04-25T07:15:56.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855374' 2024-04-25T07:15:56.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:15:56.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:15:56.639 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:15:56.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:15:56.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:15:56.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:15:56.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:15:56.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:15:56.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789965 2024-04-25T07:15:56.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789965 2024-04-25T07:15:56.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-150323855374 2-158913789965' 2024-04-25T07:15:56.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:15:56.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-150323855374 2024-04-25T07:15:56.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:15:56.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:15:56.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-150323855374 2024-04-25T07:15:56.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:15:56.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855374 2024-04-25T07:15:56.811 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 150323855374 2024-04-25T07:15:56.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 150323855374' 2024-04-25T07:15:56.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:15:57.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855371 -lt 150323855374 2024-04-25T07:15:57.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:15:58.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:15:58.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:15:58.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855374 -lt 150323855374 2024-04-25T07:15:58.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:15:58.713 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:15:58.713 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-158913789965 2024-04-25T07:15:58.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:15:58.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-158913789965 2024-04-25T07:15:58.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:15:58.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789965 2024-04-25T07:15:58.716 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 158913789965 2024-04-25T07:15:58.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 158913789965' 2024-04-25T07:15:58.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:15:59.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789965 -lt 158913789965 2024-04-25T07:15:59.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:15:59.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:15:59.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:15:59.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:15:59.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:15:59.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:15:59.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:15:59.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:15:59.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:15:59.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:15:59.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:00.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:16:00.155 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:00.155 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:00.155 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:00.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:16:00.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:16:00.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:16:00.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:16:00.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:16:00.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:16:00.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:16:00.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:00.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:00.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:00.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:00.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:00.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:00.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:01.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:16:01.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:01.175 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:01.175 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:01.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:16:01.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:16:01.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:16:01.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:16:01.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:16:01.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:16:01.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:16:01.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:16:01.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:16:02.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:16:02.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:16:02.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:16:02.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:16:02.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:16:02.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:16:02.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:02.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:02.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:02.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:02.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:02.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:02.462 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:02.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:16:02.889 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:02.889 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:02.889 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:03.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:16:03.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:16:03.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:16:03.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:16:03.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:16:03.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:16:03.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:16:03.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:16:03.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:16:04.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:16:04.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:16:04.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:16:04.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:16:04.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:16:04.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:16:04.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:04.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:04.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:04.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:04.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:04.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:04.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:04.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:16:04.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:04.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:04.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:05.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:16:05.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:16:05.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:16:05.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:16:05.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:16:05.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:16:05.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:16:05.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:16:05.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:16:05.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:16:05.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:16:05.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:16:05.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:16:05.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:16:06.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:16:06.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:06.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:06.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:06.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:06.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:06.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:06.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:07.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:16:07.210 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:07.211 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:07.211 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:07.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:16:07.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:16:07.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:16:07.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:16:07.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:16:07.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:16:07.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:16:07.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:16:07.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:16:08.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:16:08.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:16:08.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:16:08.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:16:08.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:16:09.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:16:09.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:09.920 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:09.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:09.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:09.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:09.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:09.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:10.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:16:10.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:10.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:10.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:10.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:16:10.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:16:10.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:16:10.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:16:10.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:16:10.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:16:10.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:16:10.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:16:10.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:16:11.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:16:11.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:16:11.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:16:11.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:16:11.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:16:14.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:16:14.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:14.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:14.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:14.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:14.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:14.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:14.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:15.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:16:15.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:15.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:15.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:15.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:16:15.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:16:15.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:16:15.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:16:15.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:16:15.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:16:15.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:16:15.676 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:16:15.676 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:16:16.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:16:16.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:16:16.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:16:16.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:16:16.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:16:22.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:16:22.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:22.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:22.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:22.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:22.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:22.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:22.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:23.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:16:23.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:23.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:23.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:23.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:16:23.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:16:23.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:16:23.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:16:23.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:16:23.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:16:23.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:16:23.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:16:23.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:16:24.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:16:24.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:16:24.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:16:24.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:16:24.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:16:31.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:16:31.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:16:31.451 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:16:31.451 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:16:31.451 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:16:31.451 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:16:31.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:16:31.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:16:31.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:16:31.872 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:16:31.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:16:31.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:16:32.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:16:32.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:16:32.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:16:32.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:16:32.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:16:32.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:16:32.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:16:32.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:16:32.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:16:32.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:16:32.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:16:32.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:16:32.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:16:33.495 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:16:33.495 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_fingerprint": "543d48d8-d6a1-4a5d-89cd-5ba9694a6bd9", 2024-04-25T07:16:33.495 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:16:33.495 INFO:tasks.workunit.client.0.smithi134.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:16:33.495 INFO:tasks.workunit.client.0.smithi134.stdout: "timestamp": "2024-04-25T07:16:33.498845+0000", 2024-04-25T07:16:33.495 INFO:tasks.workunit.client.0.smithi134.stdout: "tag": "", 2024-04-25T07:16:33.495 INFO:tasks.workunit.client.0.smithi134.stdout: "health": { 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "status": "HEALTH_WARN", 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "checks": { 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 17 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [], 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "OSD_DOWN": { 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:16:33.496 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 osds down", 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "osd.1 (root=default,host=smithi134) is down" 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "PG_DEGRADED": { 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 2 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.497 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:16:33.498 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: "mutes": [] 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_first_committed": 1, 2024-04-25T07:16:33.499 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_last_committed": 1, 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap": { 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:12:20.832324Z", 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:12:20.832324Z", 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release": 19, 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release_name": "squid", 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "election_strategy": 1, 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "disallowed_leaders: ": "", 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": false, 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "tiebreaker_mon": "", 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_ranks: ": "", 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "features": { 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "persistent": [ 2024-04-25T07:16:33.500 INFO:tasks.workunit.client.0.smithi134.stdout: "kraken", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "luminous", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "mimic", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap-prune", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "nautilus", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "octopus", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "pacific", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "elector-pinging", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "quincy", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "reef", 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "squid" 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "optional": [] 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "mons": [ 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.501 INFO:tasks.workunit.client.0.smithi134.stdout: "rank": 0, 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "a", 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 0 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 0, 2024-04-25T07:16:33.502 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 0, 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_location": "{}" 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "quorum": [ 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: 0 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap": { 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 40, 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:12:21.809794+0000", 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:15:17.693066+0000", 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "last_up_change": "2024-04-25T07:15:15.038151+0000", 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "last_in_change": "2024-04-25T07:15:17.193188+0000", 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_num": 5800384, 2024-04-25T07:16:33.503 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_set": [ 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "nodown", 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "noin", 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "noout", 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "pglog_hardlimit", 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snapdirs", 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "recovery_deletes", 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "sortbitwise" 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_version": 9, 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_snapshot": "", 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_max": 1, 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "max_osd": 3, 2024-04-25T07:16:33.504 INFO:tasks.workunit.client.0.smithi134.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "min_compat_client": "jewel", 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "require_osd_release": "squid", 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "allow_crimson": false, 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "pools": [ 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_name": "test", 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "create_time": "2024-04-25T07:14:14.126087+0000", 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": 1, 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_names": "hashpspool", 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 3, 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "min_size": 2, 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_rule": 0, 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:16:33.505 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "object_hash": 2, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num": 1, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num": 1, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num_target": 1, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_target": 1, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_pending": 1, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pg_merge_meta": { 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "source_pgid": "0.0", 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "ready_epoch": 0, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_started": 0, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": 0, 2024-04-25T07:16:33.506 INFO:tasks.workunit.client.0.smithi134.stdout: "source_version": "0'0", 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "target_version": "0'0" 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "last_change": "28", 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend": "0", 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "auid": 0, 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_seq": 0, 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_epoch": 0, 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_snaps": [], 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps": "[]", 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_bytes": 0, 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_objects": 0, 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "tiers": [], 2024-04-25T07:16:33.507 INFO:tasks.workunit.client.0.smithi134.stdout: "tier_of": -1, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "read_tier": -1, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "write_tier": -1, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_mode": "none", 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_bytes": 0, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_objects": 0, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_flush_age": 0, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_evict_age": 0, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profile": "", 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_params": { 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "none" 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_period": 0, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_count": 0, 2024-04-25T07:16:33.508 INFO:tasks.workunit.client.0.smithi134.stdout: "use_gmt_hitset": true, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "grade_table": [], 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "stripe_width": 0, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "expected_num_objects": 0, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "fast_read": false, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "options": {}, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "application_metadata": {}, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "read_balance": { 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "score_type": "Fair distribution", 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "score_acting": 2.9900000095367432, 2024-04-25T07:16:33.509 INFO:tasks.workunit.client.0.smithi134.stdout: "score_stable": 2.9900000095367432, 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_acting": 2, 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_stable": 2, 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity": 1, 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "osds": [ 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "c3500745-9c36-472c-b5b0-17cf5b183183", 2024-04-25T07:16:33.510 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 5, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 28, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 35, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 39, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 29, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:16:33.511 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:16:33.512 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.513 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.514 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3564868565 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6803/3564868565", 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6805/3564868565", 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3564868565", 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3564868565", 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.515 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "89b7ddae-ddc4-4e63-b12e-53901c5a0d0d", 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 0, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 10, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 23, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 30, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.516 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:16:33.517 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:16:33.518 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:16:33.519 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2462515974 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/2462515974", 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/2462515974", 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2462515974", 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2462515974", 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "exists" 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.520 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "00272bde-e741-4105-bc47-4625a8a19b02", 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 15, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 28, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 37, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 0, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 31, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.521 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.522 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:16:33.523 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.524 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3384236760 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/3384236760", 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/3384236760", 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3384236760", 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3384236760", 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.525 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_xinfo": [ 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:15:02.831126+0000", 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:12:56.907586+0000", 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.526 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:15:03.438039+0000", 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:13:33.522143+0000", 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:15:04.072862+0000", 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:16:33.527 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:14:10.149256+0000", 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap": [], 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_items": [], 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_primaries": [], 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_temp": [], 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_temp": [], 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "blocklist": {}, 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "range_blocklist": {}, 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profiles": { 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "default": { 2024-04-25T07:16:33.528 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "2", 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van" 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps_queue": [], 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "new_removed_snaps": [], 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "new_purged_snaps": [], 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_node_flags": {}, 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "device_class_flags": {}, 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": { 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_enabled": false, 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_bucket_count": 0, 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_bucket": 0 2024-04-25T07:16:33.529 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_metadata": [ 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6804/3564868565,v1:127.0.0.1:6805/3564868565]", 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:16:33.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:16:33.531 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:16:33.533 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:16:33.533 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:16:33.533 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:16:33.533 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:12:28.750502Z", 2024-04-25T07:16:33.533 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:16:33.533 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6802/3564868565,v1:127.0.0.1:6803/3564868565]", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3564868565,v1:127.0.0.1:6809/3564868565]", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3564868565,v1:127.0.0.1:6807/3564868565]", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:16:33.534 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/2462515974,v1:127.0.0.1:6813/2462515974]", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:16:33.535 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:16:33.536 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:13:04.783221Z", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/2462515974,v1:127.0.0.1:6811/2462515974]", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2462515974,v1:127.0.0.1:6817/2462515974]", 2024-04-25T07:16:33.537 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2462515974,v1:127.0.0.1:6815/2462515974]", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.538 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/3384236760,v1:127.0.0.1:6813/3384236760]", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:16:33.539 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:13:41.717511Z", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:16:33.540 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/3384236760,v1:127.0.0.1:6811/3384236760]", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3384236760,v1:127.0.0.1:6817/3384236760]", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3384236760,v1:127.0.0.1:6815/3384236760]", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:16:33.541 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_clean_epochs": { 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": { 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "per_pool": [] 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_epochs": [ 2024-04-25T07:16:33.542 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 35 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 37 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_first_committed": 1, 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_last_committed": 40, 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "crushmap": { 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": [ 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.0", 2024-04-25T07:16:33.543 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.1", 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.2", 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: "types": [ 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.544 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 0, 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd" 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "host" 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 2, 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "chassis" 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 3, 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "rack" 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.545 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 4, 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "row" 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 5, 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pdu" 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 6, 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pod" 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 7, 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "room" 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.546 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 8, 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "datacenter" 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 9, 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "zone" 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 10, 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "region" 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "root" 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.547 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "buckets": [ 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -1, 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default", 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:16:33.548 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -2, 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default~hdd", 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:16:33.549 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134", 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:16:33.550 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.551 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134~hdd", 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:16:33.552 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: "rules": [ 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 0, 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "replicated_rule", 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T07:16:33.553 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_firstn", 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.554 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "tunables": { 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_tries": 0, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_total_tries": 50, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_stable": 1, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_descents": 100, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_collision_tries": 100, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "straw_calc_version": 1, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "profile": "jewel", 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_tunables": 1, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "legacy_tunables": 0, 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "minimum_required_version": "jewel", 2024-04-25T07:16:33.555 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables": 1, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables2": 1, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v2_rules": 0, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables3": 1, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v3_rules": 0, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v4_buckets": 1, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables5": 1, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v5_rules": 0, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "has_msr_rules": 0 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_args": {} 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "fsmap": { 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "default_fscid": -1, 2024-04-25T07:16:33.556 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": { 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": {}, 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "ro_compat": {}, 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "incompat": { 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_1": "base v0.20", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_8": "no anchor table", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_9": "file layout v2", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_11": "minor log segments", 2024-04-25T07:16:33.557 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_flags": { 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "enable_multiple": true, 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "ever_enabled_multiple": true 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "standbys": [], 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "filesystems": [] 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "auth": { 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:16:33.558 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 5, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_secrets": 3 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_active": 1, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_unknown": 0, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osd": 3, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_sum": { 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 62050, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:16:33.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 101, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 101, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:16:33.560 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:16:33.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:16:33.562 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 17, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 17, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 2, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 2, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 0 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum": { 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:16:33.563 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83596, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2080, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314489204, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036944896, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2129920, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 451162, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:16:33.564 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:16:33.565 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum_by_class": { 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "hdd": { 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:16:33.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83596, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2080, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314489204, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036944896, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2129920, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 451162, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:16:33.567 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:16:33.568 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_stats": [ 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 62050, 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:16:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 101, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 101, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:16:33.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:16:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 1220608, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 185266, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:16:33.572 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 17, 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 17, 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 2, 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 2, 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 3 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_stats": [ 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 150323855381 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.573 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 42949672987 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 158913789972 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_state": [ 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "active+undersized+degraded", 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 1 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.574 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_osd": [ 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 1, 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.575 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 0, 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": [ 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": {} 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "servicemap": { 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 7, 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:16:17.327112+0000", 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "services": {} 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:16:33.576 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_last_committed": 114, 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_first_committed": 1, 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_last_committed": 445, 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout: "paxos": { 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 605, 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pn": 0, 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout: "accepted_pn": 0 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:16:33.577 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:16:33.578 INFO:tasks.workunit.client.0.smithi134.stderr:report 2557905303 2024-04-25T07:16:33.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:16:33.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2024-04-25T07:16:33.578 INFO:tasks.workunit.client.0.smithi134.stdout:delay recovery 2024-04-25T07:16:33.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:33.905 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:33.905 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:15:17.984932+0000 40'101 40:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:10:29.637545+0000 0 0 2024-04-25T07:16:33.905 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:16:33.906 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:33.906 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:16:33.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:16:33.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2024-04-25T07:16:33.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:16:33.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:16:33.922 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:16:33.923 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:33.923 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:16:33.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.0.asok 2024-04-25T07:16:33.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:16:33.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:16:34.021 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:16:34.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2024-04-25T07:16:34.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2024-04-25T07:16:34.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:16:34.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:16:34.034 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:16:34.034 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:34.034 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:16:34.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.2.asok 2024-04-25T07:16:34.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:16:34.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:16:34.142 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:16:34.154 INFO:tasks.workunit.client.0.smithi134.stdout:revive divergent 1 2024-04-25T07:16:34.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2024-04-25T07:16:34.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:34.554 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:34.554 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:15:17.984932+0000 40'101 40:126 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:10:29.637545+0000 0 0 2024-04-25T07:16:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:16:34.555 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:34.555 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:16:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2024-04-25T07:16:35.253 INFO:tasks.workunit.client.0.smithi134.stderr:noup is set 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:16:35.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:16:35.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:16:35.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:16:35.276 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:16:35.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:16:35.276 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:16:35.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:16:35.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:16:35.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:16:35.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:16:35.309 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:35.313+0000 7f0a07a997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:35.329 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:35.333+0000 7f0a07a997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:35.331 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:35.333+0000 7f0a07a997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:35.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:16:35.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2024-04-25T07:16:35.877 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:35.881+0000 7f0a07a997c0 -1 Falling back to public interface 2024-04-25T07:16:38.966 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:38.969+0000 7f0a07a997c0 -1 osd.1 28 log_to_monitors true 2024-04-25T07:16:40.786 INFO:tasks.workunit.client.0.smithi134.stdout:delay recovery divergent 2024-04-25T07:16:40.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2024-04-25T07:16:40.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:41.197 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:41.197 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:15:17.984932+0000 40'101 42:129 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:10:29.637545+0000 0 0 2024-04-25T07:16:41.197 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:16:41.197 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:41.197 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:16:41.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2024-04-25T07:16:41.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:16:41.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:16:41.215 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:16:41.215 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:41.215 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:16:41.215 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.1.asok 2024-04-25T07:16:41.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:16:41.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:16:41.315 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:16:41.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2024-04-25T07:16:42.016 INFO:tasks.workunit.client.0.smithi134.stderr:noup is unset 2024-04-25T07:16:42.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2024-04-25T07:16:42.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:42.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:16:42.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:42.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:42.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:42.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:42.037 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:16:42.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:42.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:16:42.678 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 35 up_thru 39 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/3564868565,v1:127.0.0.1:6803/3564868565] [v2:127.0.0.1:6804/3564868565,v1:127.0.0.1:6805/3564868565] exists,up c3500745-9c36-472c-b5b0-17cf5b183183 2024-04-25T07:16:42.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:16:42.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:16:42.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:16:42.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2024-04-25T07:16:42.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:42.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:16:42.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:42.680 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:16:42.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:42.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:42.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:42.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:42.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:16:43.139 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/1541961252,v1:127.0.0.1:6819/1541961252] [v2:127.0.0.1:6820/1541961252,v1:127.0.0.1:6821/1541961252] exists,up 89b7ddae-ddc4-4e63-b12e-53901c5a0d0d 2024-04-25T07:16:43.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:16:43.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:16:43.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:16:43.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2024-04-25T07:16:43.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:43.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:16:43.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:43.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:43.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:43.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:43.141 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:16:43.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:43.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:16:43.573 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 37 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/3384236760,v1:127.0.0.1:6811/3384236760] [v2:127.0.0.1:6812/3384236760,v1:127.0.0.1:6813/3384236760] exists,up 00272bde-e741-4105-bc47-4625a8a19b02 2024-04-25T07:16:43.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:16:43.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:16:43.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:16:43.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:43.982 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:43.982 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:15:17.984932+0000 40'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:10:29.637545+0000 0 0 2024-04-25T07:16:43.982 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:16:43.982 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:43.983 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:16:43.999 INFO:tasks.workunit.client.0.smithi134.stdout:wait for peering 2024-04-25T07:16:43.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2024-04-25T07:16:43.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:44.377 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:44.377 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:15:17.984932+0000 40'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:10:29.637545+0000 0 0 2024-04-25T07:16:44.377 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:16:44.377 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:44.378 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:16:44.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2024-04-25T07:16:44.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2024-04-25T07:16:44.519 INFO:tasks.workunit.client.0.smithi134.stdout:killing divergent 1 2024-04-25T07:16:44.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:44.920 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:44.920 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 62050 0 0 17 84 17 active+undersized+degraded 2024-04-25T07:15:17.984932+0000 40'101 43:131 [0,2] 0 [0,2] 0 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:10:29.637545+0000 0 0 2024-04-25T07:16:44.920 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:16:44.920 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:44.920 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:16:44.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:16:44.935 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:16:44.935 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:16:44.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:16:44.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:16:44.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:16:45.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:16:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.26105.out 2024-04-25T07:16:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.26105.out 2024-04-25T07:16:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:16:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:16:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:16:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:16:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:16:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.26105.out 2024-04-25T07:16:46.398 INFO:tasks.workunit.client.0.smithi134.stderr:Exporting 1.0 info 1.0( v 46'102 lc 0'0 (28'90,46'102] local-lis/les=45/46 n=101 ec=23/23 lis/c=45/23 les/c/f=46/24/0 sis=45) 2024-04-25T07:16:46.400 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:023dd3d2:::existing_12:head# 2024-04-25T07:16:46.400 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:04c40a0c:::existing_93:head# 2024-04-25T07:16:46.401 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:064ff5fb:::existing_36:head# 2024-04-25T07:16:46.401 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:091f9f1d:::existing_94:head# 2024-04-25T07:16:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0a865a93:::existing_54:head# 2024-04-25T07:16:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0fed513c:::existing_41:head# 2024-04-25T07:16:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:13b9e0b8:::existing_92:head# 2024-04-25T07:16:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:169157fa:::existing_81:head# 2024-04-25T07:16:46.404 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:17731382:::existing_58:head# 2024-04-25T07:16:46.405 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:18a45a39:::existing_43:head# 2024-04-25T07:16:46.405 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1c160f01:::existing_66:head# 2024-04-25T07:16:46.406 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1dd6c249:::existing_97:head# 2024-04-25T07:16:46.406 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:227e0c77:::existing_7:head# 2024-04-25T07:16:46.407 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:23e1b063:::existing_59:head# 2024-04-25T07:16:46.407 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:28566edb:::existing_78:head# 2024-04-25T07:16:46.408 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2a101182:::existing_20:head# 2024-04-25T07:16:46.408 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2d1519da:::existing_96:head# 2024-04-25T07:16:46.409 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3078b87a:::existing_86:head# 2024-04-25T07:16:46.409 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:30bde01b:::existing_15:head# 2024-04-25T07:16:46.410 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:317a7e19:::existing_56:head# 2024-04-25T07:16:46.410 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:38b9c7fa:::existing_33:head# 2024-04-25T07:16:46.411 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3e16d84b:::existing_48:head# 2024-04-25T07:16:46.411 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:45fe8203:::existing_75:head# 2024-04-25T07:16:46.412 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4e82c9ba:::existing_50:head# 2024-04-25T07:16:46.412 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5298dcdb:::existing_57:head# 2024-04-25T07:16:46.413 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:52ebaa44:::existing_32:head# 2024-04-25T07:16:46.413 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:53588912:::existing_9:head# 2024-04-25T07:16:46.414 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5b7e2de6:::existing_84:head# 2024-04-25T07:16:46.414 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5da0255c:::existing_13:head# 2024-04-25T07:16:46.415 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5da2dab8:::existing_53:head# 2024-04-25T07:16:46.415 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5ee3514b:::existing_30:head# 2024-04-25T07:16:46.415 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:602f83fe:::foo:head# 2024-04-25T07:16:46.416 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:60426cc7:::existing_62:head# 2024-04-25T07:16:46.416 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:628918ca:::existing_79:head# 2024-04-25T07:16:46.417 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:63f8ab2d:::existing_77:head# 2024-04-25T07:16:46.417 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:67e1780f:::existing_27:head# 2024-04-25T07:16:46.418 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68143ca1:::existing_71:head# 2024-04-25T07:16:46.418 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6bac9509:::existing_29:head# 2024-04-25T07:16:46.419 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6bbec798:::existing_45:head# 2024-04-25T07:16:46.419 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6cb23a40:::existing_65:head# 2024-04-25T07:16:46.420 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7055ec5e:::existing_83:head# 2024-04-25T07:16:46.420 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:70a54e2b:::existing_87:head# 2024-04-25T07:16:46.420 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:750ace21:::existing_19:head# 2024-04-25T07:16:46.421 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:78623b1d:::existing_6:head# 2024-04-25T07:16:46.421 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:79881152:::existing_63:head# 2024-04-25T07:16:46.422 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7a80c0d5:::existing_99:head# 2024-04-25T07:16:46.422 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:81490677:::existing_76:head# 2024-04-25T07:16:46.423 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:85ec8e5d:::existing_85:head# 2024-04-25T07:16:46.423 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8e2abbf0:::existing_49:head# 2024-04-25T07:16:46.424 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8ee0fe0f:::existing_52:head# 2024-04-25T07:16:46.424 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:923c5d4d:::existing_5:head# 2024-04-25T07:16:46.424 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:92731e71:::existing_89:head# 2024-04-25T07:16:46.425 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:928205cf:::existing_82:head# 2024-04-25T07:16:46.426 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:93be2267:::existing_24:head# 2024-04-25T07:16:46.426 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:99ceda2c:::existing_44:head# 2024-04-25T07:16:46.427 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9dd917ee:::existing_26:head# 2024-04-25T07:16:46.427 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a28805a6:::existing_72:head# 2024-04-25T07:16:46.428 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a294a8f6:::existing_16:head# 2024-04-25T07:16:46.428 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a2f5473a:::existing_42:head# 2024-04-25T07:16:46.429 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a504a533:::existing_90:head# 2024-04-25T07:16:46.429 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a691337e:::existing_61:head# 2024-04-25T07:16:46.430 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a6e8d7ac:::existing_28:head# 2024-04-25T07:16:46.430 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a9fde86e:::existing_18:head# 2024-04-25T07:16:46.431 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:aa309faa:::existing_68:head# 2024-04-25T07:16:46.431 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ae40b0ed:::existing_31:head# 2024-04-25T07:16:46.432 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b279b1fe:::existing_10:head# 2024-04-25T07:16:46.433 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b343e277:::existing_88:head# 2024-04-25T07:16:46.433 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b7130e90:::existing_80:head# 2024-04-25T07:16:46.433 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bc6f7309:::existing_70:head# 2024-04-25T07:16:46.434 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:be8489b7:::existing_38:head# 2024-04-25T07:16:46.435 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:beb4e5e3:::existing_47:head# 2024-04-25T07:16:46.435 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bebec877:::existing_74:head# 2024-04-25T07:16:46.435 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c04347d8:::existing_60:head# 2024-04-25T07:16:46.436 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c156033e:::existing_100:head# 2024-04-25T07:16:46.436 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c19f5a8e:::existing_8:head# 2024-04-25T07:16:46.437 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c3e6ebe4:::existing_17:head# 2024-04-25T07:16:46.437 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c6c5a7c2:::existing_39:head# 2024-04-25T07:16:46.438 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cc1d0abf:::existing_95:head# 2024-04-25T07:16:46.438 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ccb3865b:::existing_73:head# 2024-04-25T07:16:46.439 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cd84e19e:::existing_40:head# 2024-04-25T07:16:46.439 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d2cd64ce:::existing_37:head# 2024-04-25T07:16:46.440 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d41a25c9:::existing_22:head# 2024-04-25T07:16:46.440 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d7f088da:::existing_91:head# 2024-04-25T07:16:46.441 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:16:46.441 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-04-25T07:16:46.442 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd677bd3:::existing_35:head# 2024-04-25T07:16:46.443 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-04-25T07:16:46.443 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e0e22077:::existing_21:head# 2024-04-25T07:16:46.443 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e255b710:::existing_55:head# 2024-04-25T07:16:46.443 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:eff1d5db:::existing_14:head# 2024-04-25T07:16:46.444 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f1b156bf:::existing_23:head# 2024-04-25T07:16:46.444 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f5ce4199:::existing_34:head# 2024-04-25T07:16:46.445 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f6809456:::existing_11:head# 2024-04-25T07:16:46.445 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f699a2a9:::existing_67:head# 2024-04-25T07:16:46.446 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f9baac43:::existing_25:head# 2024-04-25T07:16:46.446 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fb76688e:::existing_51:head# 2024-04-25T07:16:46.447 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fbb92eec:::existing_69:head# 2024-04-25T07:16:46.447 INFO:tasks.workunit.client.0.smithi134.stderr:Export successful 2024-04-25T07:16:46.447 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T07:16:46.447 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T07:16:46.447 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T07:16:46.697 INFO:tasks.workunit.client.0.smithi134.stderr:Remove successful 2024-04-25T07:16:47.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.26105.out 2024-04-25T07:16:47.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:16:47.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:16:47.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:16:47.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:16:47.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:16:47.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.26105.out 2024-04-25T07:16:48.195 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e46 pg_nums {1={23=1}} deleted_pools ) 2024-04-25T07:16:48.195 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.0 2024-04-25T07:16:48.195 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:023dd3d2:::existing_12:head# 2024-04-25T07:16:48.195 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.275 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:04c40a0c:::existing_93:head# 2024-04-25T07:16:48.275 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.317 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:064ff5fb:::existing_36:head# 2024-04-25T07:16:48.317 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.351 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:091f9f1d:::existing_94:head# 2024-04-25T07:16:48.351 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.384 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0a865a93:::existing_54:head# 2024-04-25T07:16:48.384 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.418 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0fed513c:::existing_41:head# 2024-04-25T07:16:48.418 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.451 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:13b9e0b8:::existing_92:head# 2024-04-25T07:16:48.451 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.484 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:169157fa:::existing_81:head# 2024-04-25T07:16:48.484 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.526 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:17731382:::existing_58:head# 2024-04-25T07:16:48.526 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.560 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:18a45a39:::existing_43:head# 2024-04-25T07:16:48.560 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.593 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1c160f01:::existing_66:head# 2024-04-25T07:16:48.593 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.626 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1dd6c249:::existing_97:head# 2024-04-25T07:16:48.627 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.660 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:227e0c77:::existing_7:head# 2024-04-25T07:16:48.660 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.694 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:23e1b063:::existing_59:head# 2024-04-25T07:16:48.694 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.727 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:28566edb:::existing_78:head# 2024-04-25T07:16:48.727 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.761 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2a101182:::existing_20:head# 2024-04-25T07:16:48.761 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.794 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2d1519da:::existing_96:head# 2024-04-25T07:16:48.794 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.827 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3078b87a:::existing_86:head# 2024-04-25T07:16:48.827 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.861 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:30bde01b:::existing_15:head# 2024-04-25T07:16:48.861 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.895 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:317a7e19:::existing_56:head# 2024-04-25T07:16:48.895 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.944 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:38b9c7fa:::existing_33:head# 2024-04-25T07:16:48.945 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:48.978 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3e16d84b:::existing_48:head# 2024-04-25T07:16:48.978 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.011 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:45fe8203:::existing_75:head# 2024-04-25T07:16:49.012 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.045 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4e82c9ba:::existing_50:head# 2024-04-25T07:16:49.045 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.078 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5298dcdb:::existing_57:head# 2024-04-25T07:16:49.079 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.112 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:52ebaa44:::existing_32:head# 2024-04-25T07:16:49.112 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.145 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:53588912:::existing_9:head# 2024-04-25T07:16:49.145 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.179 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5b7e2de6:::existing_84:head# 2024-04-25T07:16:49.179 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.212 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5da0255c:::existing_13:head# 2024-04-25T07:16:49.212 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.245 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5da2dab8:::existing_53:head# 2024-04-25T07:16:49.246 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.282 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5ee3514b:::existing_30:head# 2024-04-25T07:16:49.282 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.312 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:602f83fe:::foo:head# 2024-04-25T07:16:49.312 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.346 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:60426cc7:::existing_62:head# 2024-04-25T07:16:49.346 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.379 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:628918ca:::existing_79:head# 2024-04-25T07:16:49.379 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.446 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:63f8ab2d:::existing_77:head# 2024-04-25T07:16:49.446 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.496 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67e1780f:::existing_27:head# 2024-04-25T07:16:49.497 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.538 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68143ca1:::existing_71:head# 2024-04-25T07:16:49.538 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.580 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6bac9509:::existing_29:head# 2024-04-25T07:16:49.580 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.622 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6bbec798:::existing_45:head# 2024-04-25T07:16:49.622 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.664 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6cb23a40:::existing_65:head# 2024-04-25T07:16:49.664 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.706 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7055ec5e:::existing_83:head# 2024-04-25T07:16:49.706 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.747 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:70a54e2b:::existing_87:head# 2024-04-25T07:16:49.747 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.789 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:750ace21:::existing_19:head# 2024-04-25T07:16:49.789 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.831 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:78623b1d:::existing_6:head# 2024-04-25T07:16:49.831 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.873 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:79881152:::existing_63:head# 2024-04-25T07:16:49.873 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.940 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7a80c0d5:::existing_99:head# 2024-04-25T07:16:49.940 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:49.998 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:81490677:::existing_76:head# 2024-04-25T07:16:49.998 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.082 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:85ec8e5d:::existing_85:head# 2024-04-25T07:16:50.082 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.124 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8e2abbf0:::existing_49:head# 2024-04-25T07:16:50.124 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.165 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8ee0fe0f:::existing_52:head# 2024-04-25T07:16:50.166 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.207 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:923c5d4d:::existing_5:head# 2024-04-25T07:16:50.208 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.249 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:92731e71:::existing_89:head# 2024-04-25T07:16:50.249 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.291 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:928205cf:::existing_82:head# 2024-04-25T07:16:50.291 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.333 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:93be2267:::existing_24:head# 2024-04-25T07:16:50.333 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.374 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:99ceda2c:::existing_44:head# 2024-04-25T07:16:50.375 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.416 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9dd917ee:::existing_26:head# 2024-04-25T07:16:50.416 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.458 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a28805a6:::existing_72:head# 2024-04-25T07:16:50.458 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.500 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a294a8f6:::existing_16:head# 2024-04-25T07:16:50.500 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.542 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a2f5473a:::existing_42:head# 2024-04-25T07:16:50.542 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.583 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a504a533:::existing_90:head# 2024-04-25T07:16:50.584 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.626 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a691337e:::existing_61:head# 2024-04-25T07:16:50.626 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.667 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a6e8d7ac:::existing_28:head# 2024-04-25T07:16:50.667 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.709 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a9fde86e:::existing_18:head# 2024-04-25T07:16:50.709 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.751 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:aa309faa:::existing_68:head# 2024-04-25T07:16:50.751 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.792 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ae40b0ed:::existing_31:head# 2024-04-25T07:16:50.793 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.868 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b279b1fe:::existing_10:head# 2024-04-25T07:16:50.868 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.909 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b343e277:::existing_88:head# 2024-04-25T07:16:50.910 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.952 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b7130e90:::existing_80:head# 2024-04-25T07:16:50.952 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:50.993 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bc6f7309:::existing_70:head# 2024-04-25T07:16:50.993 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.035 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:be8489b7:::existing_38:head# 2024-04-25T07:16:51.035 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.077 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:beb4e5e3:::existing_47:head# 2024-04-25T07:16:51.077 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.110 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bebec877:::existing_74:head# 2024-04-25T07:16:51.110 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.152 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c04347d8:::existing_60:head# 2024-04-25T07:16:51.152 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.186 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c156033e:::existing_100:head# 2024-04-25T07:16:51.186 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.219 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c19f5a8e:::existing_8:head# 2024-04-25T07:16:51.219 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.252 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c3e6ebe4:::existing_17:head# 2024-04-25T07:16:51.252 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.286 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c6c5a7c2:::existing_39:head# 2024-04-25T07:16:51.286 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.319 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cc1d0abf:::existing_95:head# 2024-04-25T07:16:51.319 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.353 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ccb3865b:::existing_73:head# 2024-04-25T07:16:51.353 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.386 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cd84e19e:::existing_40:head# 2024-04-25T07:16:51.387 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.453 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d2cd64ce:::existing_37:head# 2024-04-25T07:16:51.454 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.503 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d41a25c9:::existing_22:head# 2024-04-25T07:16:51.504 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.537 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d7f088da:::existing_91:head# 2024-04-25T07:16:51.537 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.571 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:16:51.571 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.604 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-04-25T07:16:51.604 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.637 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd677bd3:::existing_35:head# 2024-04-25T07:16:51.638 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.671 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-04-25T07:16:51.671 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.704 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e0e22077:::existing_21:head# 2024-04-25T07:16:51.704 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.738 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e255b710:::existing_55:head# 2024-04-25T07:16:51.738 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.771 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:eff1d5db:::existing_14:head# 2024-04-25T07:16:51.771 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.804 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f1b156bf:::existing_23:head# 2024-04-25T07:16:51.805 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.838 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f5ce4199:::existing_34:head# 2024-04-25T07:16:51.838 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.872 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f6809456:::existing_11:head# 2024-04-25T07:16:51.872 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.939 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f699a2a9:::existing_67:head# 2024-04-25T07:16:51.939 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:51.997 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f9baac43:::existing_25:head# 2024-04-25T07:16:51.997 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:52.064 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fb76688e:::existing_51:head# 2024-04-25T07:16:52.064 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:52.106 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fbb92eec:::existing_69:head# 2024-04-25T07:16:52.106 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:16:52.148 INFO:tasks.workunit.client.0.smithi134.stdout:write_pg epoch 46 info 1.0( v 46'102 lc 0'0 (28'90,46'102] local-lis/les=45/46 n=101 ec=23/23 lis/c=45/23 les/c/f=46/24/0 sis=45) 2024-04-25T07:16:52.189 INFO:tasks.workunit.client.0.smithi134.stdout:Import successful 2024-04-25T07:16:52.727 INFO:tasks.workunit.client.0.smithi134.stdout:reviving divergent 1 2024-04-25T07:16:52.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2024-04-25T07:16:52.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:16:53.134 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:16:53.134 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 4 4 0 0 62050 0 0 11 90 11 active+recovering+degraded 2024-04-25T07:16:43.622385+0000 40'101 46:130 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:16:53.134 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:16:53.134 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:16:53.134 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:16:53.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2024-04-25T07:16:53.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:16:53.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:16:53.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:16:53.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:16:53.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:16:53.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:16:53.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:16:53.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:16:53.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:16:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:16:53.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:16:53.156 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:16:53.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:16:53.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:16:53.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:16:53.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:16:53.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:16:53.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:16:53.188 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:53.193+0000 7f70c2b867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:53.190 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:53.193+0000 7f70c2b867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:53.191 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:53.197+0000 7f70c2b867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:16:53.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:16:53.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:53.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:16:53.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:53.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:53.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:53.575 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:16:53.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:53.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:53.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:16:53.744 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:53.749+0000 7f70c2b867c0 -1 Falling back to public interface 2024-04-25T07:16:54.052 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/1541961252,v1:127.0.0.1:6819/1541961252] [v2:127.0.0.1:6820/1541961252,v1:127.0.0.1:6821/1541961252] exists,up 89b7ddae-ddc4-4e63-b12e-53901c5a0d0d 2024-04-25T07:16:54.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:16:54.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:16:54.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:16:54.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2024-04-25T07:16:54.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:16:54.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:16:54.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:16:54.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:16:54.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:16:54.055 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:16:54.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:16:54.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:16:54.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:16:54.464 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 45 up_thru 45 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/1541961252,v1:127.0.0.1:6819/1541961252] [v2:127.0.0.1:6820/1541961252,v1:127.0.0.1:6821/1541961252] exists,up 89b7ddae-ddc4-4e63-b12e-53901c5a0d0d 2024-04-25T07:16:54.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:16:54.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:16:54.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:16:54.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2024-04-25T07:16:56.768 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:16:56.773+0000 7f70c2b867c0 -1 osd.1 46 log_to_monitors true 2024-04-25T07:17:14.468 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2024-04-25T07:17:14.468 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:17:14.468 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:17:14.468 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:17:14.468 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:14.468 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:14.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.1.asok 2024-04-25T07:17:14.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2024-04-25T07:17:14.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T07:17:14.584 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:17:14.584 INFO:tasks.workunit.client.0.smithi134.stdout: "ops": [], 2024-04-25T07:17:14.584 INFO:tasks.workunit.client.0.smithi134.stdout: "num_ops": 0 2024-04-25T07:17:14.584 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:17:14.596 INFO:tasks.workunit.client.0.smithi134.stdout:allowing recovery 2024-04-25T07:17:14.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2024-04-25T07:17:14.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:17:15.004 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:17:15.004 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-04-25T07:16:58.881965+0000 46'102 51:153 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:43:27.449934+0000 0 0 2024-04-25T07:17:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:17:15.005 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:17:15.005 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:17:15.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:17:15.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:17:15.173 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:17:15.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:17:15.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:17:15.336 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:17:15.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2024-04-25T07:17:15.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:17:15.494 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:17:15.507 INFO:tasks.workunit.client.0.smithi134.stdout:reading divergent objects 2024-04-25T07:17:15.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2024-04-25T07:17:15.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2024-04-25T07:17:15.893 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:17:15.893 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 0 0 0 62667 0 0 12 90 12 active+clean 2024-04-25T07:16:58.881965+0000 46'102 51:153 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:14:14.176338+0000 0'0 2024-04-25T07:14:14.176338+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:43:27.449934+0000 0 0 2024-04-25T07:17:15.893 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:17:15.893 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:17:15.894 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:17:15.910 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2024-04-25T07:17:15.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2024-04-25T07:17:15.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:17:15.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:17:15.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:17:15.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:17:15.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:17:15.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:17:16.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:17:16.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:17:16.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2024-04-25T07:17:16.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:17:16.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2024-04-25T07:17:16.100 INFO:tasks.workunit.client.0.smithi134.stderr:error getting test/newobject_1: (2) No such file or directory 2024-04-25T07:17:16.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:17:16.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2024-04-25T07:17:16.135 INFO:tasks.workunit.client.0.smithi134.stderr:error getting test/newobject_2: (2) No such file or directory 2024-04-25T07:17:16.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2024-04-25T07:17:16.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:17:16.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4637.0:1 2024-04-25T07:14:51.998687+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: keeping 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4637.0:1 2024-04-25T07:14:51.998687+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'106 last_divergent_update: 28'106 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: more recent entry found: 40'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4754.0:1 2024-04-25T07:15:55.778708+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4633.0:1 2024-04-25T07:14:51.994096+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: keeping 28'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4633.0:1 2024-04-25T07:14:51.994096+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'104 last_divergent_update: 28'104 2024-04-25T07:17:16.143 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4636.0:1 2024-04-25T07:14:51.997057+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: keeping 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4636.0:1 2024-04-25T07:14:51.997057+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'105 last_divergent_update: 28'105 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4636.0:1 2024-04-25T07:14:51.997057+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2024-04-25T07:17:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4624.0:1 2024-04-25T07:14:51.981709+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: keeping 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4624.0:1 2024-04-25T07:14:51.981709+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'102 last_divergent_update: 28'102 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'102 (28'2) modify 1:97db2a0f:::existing_2:head by client.4624.0:1 2024-04-25T07:14:51.981709+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'101 (28'1) modify 1:d858c035:::existing_1:head by client.4625.0:1 2024-04-25T07:14:51.981278+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:17:16.145 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: keeping 28'101 (28'1) modify 1:d858c035:::existing_1:head by client.4625.0:1 2024-04-25T07:14:51.981278+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'101 last_divergent_update: 28'101 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'101 (28'1) modify 1:d858c035:::existing_1:head by client.4625.0:1 2024-04-25T07:14:51.981278+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4629.0:1 2024-04-25T07:14:51.989715+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:17:16.146 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 20 _merge_object_divergent_entries: keeping 28'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4629.0:1 2024-04-25T07:14:51.989715+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:17:16.147 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:17:16.147 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'103 last_divergent_update: 28'103 2024-04-25T07:17:16.147 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-04-25T07:17:16.147 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:16:42.581+0000 7f09eba23640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-04-25T07:17:16.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:17:16.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:17:16.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2024-04-25T07:17:16.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:17:16.155 INFO:tasks.workunit.client.0.smithi134.stdout:success 2024-04-25T07:17:16.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2024-04-25T07:17:16.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.26105.out 2024-04-25T07:17:16.156 INFO:tasks.workunit.client.0.smithi134.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.26105.out': No such file or directory 2024-04-25T07:17:16.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2024-04-25T07:17:16.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:17:16.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:17:16.662 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T07:17:16.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2024-04-25T07:17:16.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:17:16.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:17:16.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:17:16.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:17:16.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:17:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:17:16.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:17:16.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:17:16.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:17:16.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:17:16.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:17:16.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:17:16.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:17:16.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:17:16.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:17:16.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:17:16.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:17:16.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:17:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:17:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:17:16.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:17:16.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:16.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:16.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26105 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:17:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:17:16.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:17:16.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:17:16.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:17:16.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:17:16.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:17:16.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:17:16.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:17:16.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:17:16.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:17:16.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:17:16.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:17:16.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:17:16.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:17:16.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:17:16.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:17:16.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:17:16.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:16.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:16.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26105 2024-04-25T07:17:16.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:17:16.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:17:16.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:17:16.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:17:16.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:16.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:16.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26105 2024-04-25T07:17:16.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:17:16.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:17:16.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:17:16.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:17:16.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2024-04-25T07:17:16.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2024-04-25T07:17:16.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2024-04-25T07:17:16.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2024-04-25T07:17:16.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2024-04-25T07:17:16.836 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2024-04-25T07:17:16.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2024-04-25T07:17:16.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2024-04-25T07:17:16.837 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:17:16.837 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:17:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2024-04-25T07:17:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:17:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:17:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:17:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:17:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:17:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:17:17.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:17:17.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:17:17.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:17:17.312 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:17:17.312 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:17.312 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:17.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:17:17.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:17:17.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:17:17.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:17:17.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:17:17.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:17:17.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:17:17.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:17:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:17:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:17:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:17:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:17:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-mon.a.asok 2024-04-25T07:17:17.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:17:17.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26105/ceph-mon.a.asok config get fsid 2024-04-25T07:17:17.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:17:17.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:17:17.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:17:17.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:17:17.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:17:17.647 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:17:17.647 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:17:17.647 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:17:17.648 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:17:17.648 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:17.648 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-mon.a.asok 2024-04-25T07:17:17.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:17:17.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26105/ceph-mon.a.asok config get mon_host 2024-04-25T07:17:17.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2024-04-25T07:17:17.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:17:17.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:17:17.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:17:17.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:17:17.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:17:17.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:17:18.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:17:18.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:17:18.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:17:18.651 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:17:18.652 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:18.652 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:18.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:17:18.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:17:18.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:17:18.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:17:18.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:17:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:17:18.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:17:18.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=690eaf5b-9dca-4c84-9e79-ba2565d401e0 2024-04-25T07:17:18.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 690eaf5b-9dca-4c84-9e79-ba2565d401e0' 2024-04-25T07:17:18.725 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 690eaf5b-9dca-4c84-9e79-ba2565d401e0 2024-04-25T07:17:18.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:17:18.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+AypmgOP8LBAA96PcxZA+gaydhPKJIxzFGg== 2024-04-25T07:17:18.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+AypmgOP8LBAA96PcxZA+gaydhPKJIxzFGg=="}' 2024-04-25T07:17:18.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 690eaf5b-9dca-4c84-9e79-ba2565d401e0 -i td/divergent-priors/0/new.json 2024-04-25T07:17:19.050 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:17:19.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:17:19.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+AypmgOP8LBAA96PcxZA+gaydhPKJIxzFGg== --osd-uuid 690eaf5b-9dca-4c84-9e79-ba2565d401e0 2024-04-25T07:17:19.105 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:19.108+0000 7f06a5fbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:19.119 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:19.124+0000 7f06a5fbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:19.120 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:19.124+0000 7f06a5fbe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:19.121 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:19.124+0000 7f06a5fbe7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:17:19.121 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:19.124+0000 7f06a5fbe7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:17:19.121 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:19.124+0000 7f06a5fbe7c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:17:19.158 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:19.160+0000 7f06a5fbe7c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:17:25.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:17:25.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:17:25.613 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:17:25.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:17:25.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:17:26.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:17:26.165 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:17:26.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:17:26.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:17:26.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:17:26.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:17:26.205 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:26.208+0000 7f815282e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:26.240 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:26.244+0000 7f815282e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:26.242 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:26.244+0000 7f815282e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:26.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:17:26.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:17:26.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:17:26.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:17:26.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:17:26.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:26.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:17:26.596 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:17:26.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:26.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:26.789 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:26.792+0000 7f815282e7c0 -1 Falling back to public interface 2024-04-25T07:17:27.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:28.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:28.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:28.031 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:17:28.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:17:28.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:28.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:28.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:29.420 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:29.424+0000 7f815282e7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:17:29.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:29.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:29.446 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:17:29.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:17:29.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:29.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:30.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:31.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:31.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:31.145 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:17:31.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:17:31.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:31.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:31.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:32.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:32.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:32.698 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:17:32.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:17:32.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:32.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:33.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:34.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:34.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:34.260 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:17:34.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:17:34.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:34.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:34.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:35.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:35.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:35.708 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:17:35.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:17:35.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:35.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:36.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:37.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:37.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:37.262 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:17:37.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:17:37.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:37.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:37.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:38.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:38.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:38.724 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:17:38.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:17:38.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:38.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:39.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:40.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:40.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:40.283 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:17:40.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:17:40.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:40.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:40.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:41.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:41.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:41.722 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:17:41.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:17:41.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:41.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:42.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:43.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:43.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:43.188 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:17:43.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:17:43.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:43.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:44.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:45.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:45.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:45.269 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:17:45.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:17:45.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:45.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:46.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:47.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:47.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:47.224 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:17:47.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:17:47.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:47.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:48.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:49.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:49.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:49.189 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:17:49.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:17:49.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:49.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:50.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:51.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:51.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:51.743 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:17:51.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:17:51.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:51.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:52.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:17:53.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:17:53.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:17:53.624 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:17:53.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:17:53.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:17:53.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:17:54.061 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1448708081,v1:127.0.0.1:6803/1448708081] [v2:127.0.0.1:6804/1448708081,v1:127.0.0.1:6805/1448708081] exists,up 690eaf5b-9dca-4c84-9e79-ba2565d401e0 2024-04-25T07:17:54.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:17:54.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:17:54.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:17:54.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:17:54.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2024-04-25T07:17:54.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:17:54.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:17:54.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:17:54.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:17:54.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:17:54.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:17:54.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:17:54.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:17:54.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:17:54.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:17:54.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8ebabd29-af92-4d13-bc6b-3f0ab12e5b80 2024-04-25T07:17:54.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 8ebabd29-af92-4d13-bc6b-3f0ab12e5b80' 2024-04-25T07:17:54.069 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 8ebabd29-af92-4d13-bc6b-3f0ab12e5b80 2024-04-25T07:17:54.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:17:54.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiAypmpVcBBhAAT/kSVJiU7FB+3p/T2BiWCQ== 2024-04-25T07:17:54.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiAypmpVcBBhAAT/kSVJiU7FB+3p/T2BiWCQ=="}' 2024-04-25T07:17:54.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8ebabd29-af92-4d13-bc6b-3f0ab12e5b80 -i td/divergent-priors/1/new.json 2024-04-25T07:17:54.603 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:17:54.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:17:54.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiAypmpVcBBhAAT/kSVJiU7FB+3p/T2BiWCQ== --osd-uuid 8ebabd29-af92-4d13-bc6b-3f0ab12e5b80 2024-04-25T07:17:54.654 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:54.656+0000 7ff8b52537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:54.689 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:54.692+0000 7ff8b52537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:54.690 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:54.696+0000 7ff8b52537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:17:54.691 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:54.696+0000 7ff8b52537c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:17:54.691 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:54.696+0000 7ff8b52537c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:17:54.691 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:54.696+0000 7ff8b52537c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:17:54.729 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:17:54.732+0000 7ff8b52537c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:18:01.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:18:01.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:18:01.381 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:18:01.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:18:01.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:18:01.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:18:01.940 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:18:01.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:18:01.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:18:01.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:18:01.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:18:01.979 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:01.984+0000 7ff8f73147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:01.981 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:01.984+0000 7ff8f73147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:01.983 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:01.988+0000 7ff8f73147c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:02.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:18:02.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:02.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:18:02.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:02.344 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:18:02.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:02.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:02.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:02.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:02.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:02.552 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:02.556+0000 7ff8f73147c0 -1 Falling back to public interface 2024-04-25T07:18:02.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:03.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:03.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:03.780 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:18:03.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:18:03.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:03.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:04.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:05.191 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:05.196+0000 7ff8f73147c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:18:05.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:05.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:05.206 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:18:05.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:18:05.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:05.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:05.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:06.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:06.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:06.953 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:18:06.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:18:06.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:06.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:07.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:08.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:08.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:08.383 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:18:08.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:18:08.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:08.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:08.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:09.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:09.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:09.812 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:18:09.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:18:09.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:09.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:10.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:11.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:11.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:11.421 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:18:11.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:18:11.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:11.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:11.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:12.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:12.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:12.877 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:18:12.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:18:12.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:12.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:13.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:14.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:14.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:14.394 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:18:14.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:18:14.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:14.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:14.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:15.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:15.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:15.852 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:18:15.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:18:15.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:15.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:16.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:17.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:17.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:17.395 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:18:17.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:18:17.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:17.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:18.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:19.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:19.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:19.012 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:18:19.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:18:19.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:19.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:19.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:20.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:20.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:20.589 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:18:20.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:18:20.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:20.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:21.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:22.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:22.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:22.495 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:18:22.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:18:22.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:22.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:23.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:24.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:24.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:24.320 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:18:24.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:18:24.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:24.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:24.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:25.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:25.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:25.908 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:18:25.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:18:25.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:25.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:26.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:27.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:27.627 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:18:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:18:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:27.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:28.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:29.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:29.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:29.553 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:18:29.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:18:29.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:29.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:18:30.010 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3928262223,v1:127.0.0.1:6811/3928262223] [v2:127.0.0.1:6812/3928262223,v1:127.0.0.1:6813/3928262223] exists,up 8ebabd29-af92-4d13-bc6b-3f0ab12e5b80 2024-04-25T07:18:30.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:18:30.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:18:30.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:18:30.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:18:30.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:18:30.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:18:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:18:30.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:18:30.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c427ab00-4be7-4143-8d54-955f5dac6f79 2024-04-25T07:18:30.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c427ab00-4be7-4143-8d54-955f5dac6f79' 2024-04-25T07:18:30.019 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 c427ab00-4be7-4143-8d54-955f5dac6f79 2024-04-25T07:18:30.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:18:30.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDGAypmfW8gAxAAn9axwRtYUuv58bqAaYVv+g== 2024-04-25T07:18:30.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDGAypmfW8gAxAAn9axwRtYUuv58bqAaYVv+g=="}' 2024-04-25T07:18:30.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c427ab00-4be7-4143-8d54-955f5dac6f79 -i td/divergent-priors/2/new.json 2024-04-25T07:18:30.661 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:18:30.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:18:30.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDGAypmfW8gAxAAn9axwRtYUuv58bqAaYVv+g== --osd-uuid c427ab00-4be7-4143-8d54-955f5dac6f79 2024-04-25T07:18:30.720 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:30.723+0000 7f206fa197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:30.722 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:30.727+0000 7f206fa197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:30.723 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:30.727+0000 7f206fa197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:30.724 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:30.727+0000 7f206fa197c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:18:30.724 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:30.727+0000 7f206fa197c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:18:30.724 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:30.727+0000 7f206fa197c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:18:30.767 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:30.771+0000 7f206fa197c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:18:37.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:18:37.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:18:37.682 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:18:37.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:18:37.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:18:38.250 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:18:38.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:18:38.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:18:38.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:18:38.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:18:38.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:18:38.290 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:38.291+0000 7ff0a9ba97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:38.316 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:38.319+0000 7ff0a9ba97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:38.317 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:38.319+0000 7ff0a9ba97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:18:38.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:18:38.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:18:38.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:18:38.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:18:38.668 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:18:38.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:18:38.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:38.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:18:38.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:38.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:38.884 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:38.887+0000 7ff0a9ba97c0 -1 Falling back to public interface 2024-04-25T07:18:39.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:40.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:40.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:40.099 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:18:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:18:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:40.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:41.517 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:18:41.519+0000 7ff0a9ba97c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:18:41.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:41.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:41.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:18:41.518 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:18:41.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:41.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:42.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:43.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:43.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:43.217 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:18:43.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:18:43.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:43.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:43.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:44.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:44.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:44.657 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:18:44.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:18:44.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:44.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:45.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:46.133 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:18:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:18:46.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:46.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:46.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:47.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:47.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:47.661 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:18:47.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:18:47.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:47.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:48.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:49.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:49.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:49.218 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:18:49.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:18:49.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:49.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:49.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:50.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:50.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:50.724 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:18:50.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:18:50.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:50.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:51.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:52.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:52.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:52.152 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:18:52.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:18:52.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:52.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:52.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:53.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:53.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:53.645 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:18:53.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:18:53.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:53.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:54.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:55.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:55.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:55.212 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:18:55.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:18:55.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:55.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:55.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:56.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:56.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:56.749 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:18:56.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:18:56.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:56.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:57.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:18:58.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:18:58.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:18:58.218 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:18:58.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:18:58.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:18:58.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:18:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:00.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:00.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:00.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:19:00.028 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:19:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:01.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:02.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:02.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:02.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:19:02.179 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:19:02.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:02.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:02.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:03.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:03.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:03.639 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:19:03.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:19:03.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:03.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:04.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:05.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:05.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:05.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:19:05.855 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:19:05.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:05.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:06.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:19:07.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:19:07.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:19:07.698 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:19:07.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:19:07.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:19:07.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:19:08.184 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3310210323,v1:127.0.0.1:6819/3310210323] [v2:127.0.0.1:6820/3310210323,v1:127.0.0.1:6821/3310210323] exists,up c427ab00-4be7-4143-8d54-955f5dac6f79 2024-04-25T07:19:08.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:19:08.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:19:08.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:19:08.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2024-04-25T07:19:08.849 INFO:tasks.workunit.client.0.smithi134.stderr:noout is set 2024-04-25T07:19:08.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2024-04-25T07:19:09.786 INFO:tasks.workunit.client.0.smithi134.stderr:noin is set 2024-04-25T07:19:09.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2024-04-25T07:19:10.590 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T07:19:10.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2024-04-25T07:19:10.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T07:19:11.284 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T07:19:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:19:12.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2024-04-25T07:19:13.103 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 3 2024-04-25T07:19:13.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2024-04-25T07:19:13.999 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 min_size to 2 2024-04-25T07:19:14.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2024-04-25T07:19:14.870 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 pg_autoscale_mode to on 2024-04-25T07:19:14.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: divergent=-1 2024-04-25T07:19:14.893 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: date +%s 2024-04-25T07:19:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: start_time=1714029554 2024-04-25T07:19:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:655: TEST_divergent_3: max_duration=300 2024-04-25T07:19:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' -1 -le -1 ']' 2024-04-25T07:19:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: flush_pg_stats 2024-04-25T07:19:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:19:14.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:19:15.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:19:15.381 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:19:15.381 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:19:15.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:19:15.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:15.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:19:15.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836506 2024-04-25T07:19:15.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836506 2024-04-25T07:19:15.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506' 2024-04-25T07:19:15.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:15.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:19:15.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672973 2024-04-25T07:19:15.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672973 2024-04-25T07:19:15.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672973' 2024-04-25T07:19:15.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:15.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:19:15.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509449 2024-04-25T07:19:15.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509449 2024-04-25T07:19:15.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836506 1-42949672973 2-64424509449' 2024-04-25T07:19:15.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:15.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836506 2024-04-25T07:19:15.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:15.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:19:15.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836506 2024-04-25T07:19:15.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:15.887 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836506 2024-04-25T07:19:15.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836506 2024-04-25T07:19:15.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836506' 2024-04-25T07:19:15.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:19:16.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836506 2024-04-25T07:19:16.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:19:17.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:19:17.276 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:19:17.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836506 2024-04-25T07:19:17.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:17.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672973 2024-04-25T07:19:17.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:17.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:19:17.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672973 2024-04-25T07:19:17.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:17.716 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672973 2024-04-25T07:19:17.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672973 2024-04-25T07:19:17.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2024-04-25T07:19:17.716 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:19:18.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672973 -lt 42949672973 2024-04-25T07:19:18.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:18.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509449 2024-04-25T07:19:18.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:18.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:19:18.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509449 2024-04-25T07:19:18.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:18.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509449 2024-04-25T07:19:18.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-04-25T07:19:18.143 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509449 2024-04-25T07:19:18.143 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:19:18.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509449 2024-04-25T07:19:18.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: wait_for_clean 2024-04-25T07:19:18.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:19:18.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:19:18.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:19:18.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:19:18.552 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:19:18.552 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:19:18.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:19:18.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:19:18.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:19:18.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:19:18.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:19:18.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:19:18.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:19:18.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:19:18.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:19:19.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:19:19.067 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:19:19.068 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:19:19.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:19:19.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:19.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:19:19.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836509 2024-04-25T07:19:19.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836509 2024-04-25T07:19:19.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509' 2024-04-25T07:19:19.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:19.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:19:19.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672975 2024-04-25T07:19:19.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672975 2024-04-25T07:19:19.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672975' 2024-04-25T07:19:19.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:19.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:19:19.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509452 2024-04-25T07:19:19.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509452 2024-04-25T07:19:19.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672975 2-64424509452' 2024-04-25T07:19:19.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:19.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836509 2024-04-25T07:19:19.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:19.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:19:19.577 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-04-25T07:19:19.577 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:19.578 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836509 2024-04-25T07:19:19.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836509 2024-04-25T07:19:19.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-04-25T07:19:19.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:19:20.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836509 2024-04-25T07:19:20.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:20.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672975 2024-04-25T07:19:20.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:20.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:19:20.021 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672975 2024-04-25T07:19:20.021 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:20.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672975 2024-04-25T07:19:20.022 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672975 2024-04-25T07:19:20.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2024-04-25T07:19:20.023 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:19:20.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672973 -lt 42949672975 2024-04-25T07:19:20.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:19:21.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:19:21.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:19:21.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672975 2024-04-25T07:19:21.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:21.889 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509452 2024-04-25T07:19:21.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:21.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:19:21.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509452 2024-04-25T07:19:21.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:21.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509452 2024-04-25T07:19:21.894 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509452 2024-04-25T07:19:21.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-04-25T07:19:21.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:19:22.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509452 -lt 64424509452 2024-04-25T07:19:22.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:19:22.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:22.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:22.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:19:22.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:19:22.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:19:22.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:19:22.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:19:22.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:19:22.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:19:22.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:19:23.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:19:23.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:19:23.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:23.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:23.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:19:23.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:19:23.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:19:23.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2024-04-25T07:19:23.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: ceph pg dump pgs --format=json 2024-04-25T07:19:24.413 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:19:24.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: divergent=1 2024-04-25T07:19:24.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2024-04-25T07:19:24.430 INFO:tasks.workunit.client.0.smithi134.stdout:primary and soon to be divergent is 1 2024-04-25T07:19:24.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:665: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:19:24.814 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:19:24.814 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:19:12.359301+0000 0'0 31:25 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:19:11.050015+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:31:16.449013+0000 0 0 2024-04-25T07:19:24.815 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:19:24.815 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:19:24.815 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:19:24.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: date +%s 2024-04-25T07:19:24.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:667: TEST_divergent_3: current_time=1714029564 2024-04-25T07:19:24.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: expr 1714029564 - 1714029554 2024-04-25T07:19:24.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:668: TEST_divergent_3: elapsed_time=10 2024-04-25T07:19:24.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: '[' 10 -gt 300 ']' 2024-04-25T07:19:24.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: '[' 1 -le -1 ']' 2024-04-25T07:19:24.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:675: TEST_divergent_3: local non_divergent= 2024-04-25T07:19:24.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:19:24.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 0 = 1 ']' 2024-04-25T07:19:24.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0' 2024-04-25T07:19:24.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:19:24.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 1 = 1 ']' 2024-04-25T07:19:24.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:679: TEST_divergent_3: continue 2024-04-25T07:19:24.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: for i in $osds 2024-04-25T07:19:24.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: '[' 2 = 1 ']' 2024-04-25T07:19:24.835 INFO:tasks.workunit.client.0.smithi134.stdout:writing initial objects 2024-04-25T07:19:24.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: non_divergent=' 0 2' 2024-04-25T07:19:24.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:684: TEST_divergent_3: echo 'writing initial objects' 2024-04-25T07:19:24.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: seq 1 100 2024-04-25T07:19:24.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:24.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2024-04-25T07:19:25.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:25.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2024-04-25T07:19:25.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:25.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2024-04-25T07:19:25.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:25.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2024-04-25T07:19:25.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:25.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2024-04-25T07:19:26.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:26.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2024-04-25T07:19:27.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:27.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2024-04-25T07:19:27.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:27.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2024-04-25T07:19:27.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:27.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2024-04-25T07:19:27.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:27.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2024-04-25T07:19:27.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:27.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2024-04-25T07:19:27.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:27.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2024-04-25T07:19:28.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:28.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2024-04-25T07:19:28.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:28.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2024-04-25T07:19:28.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:28.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2024-04-25T07:19:28.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:28.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2024-04-25T07:19:28.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:28.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2024-04-25T07:19:29.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:29.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2024-04-25T07:19:29.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:29.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2024-04-25T07:19:29.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:29.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2024-04-25T07:19:29.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:29.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2024-04-25T07:19:29.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:29.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2024-04-25T07:19:30.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:30.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2024-04-25T07:19:30.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:30.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2024-04-25T07:19:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2024-04-25T07:19:30.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:30.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2024-04-25T07:19:30.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:30.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2024-04-25T07:19:31.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:31.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2024-04-25T07:19:31.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:31.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2024-04-25T07:19:31.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:31.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2024-04-25T07:19:32.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:32.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2024-04-25T07:19:32.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:32.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2024-04-25T07:19:33.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:33.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2024-04-25T07:19:33.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:33.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2024-04-25T07:19:34.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:34.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2024-04-25T07:19:34.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:34.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2024-04-25T07:19:34.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:34.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2024-04-25T07:19:35.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:35.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2024-04-25T07:19:35.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:35.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2024-04-25T07:19:35.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:35.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2024-04-25T07:19:35.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:35.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2024-04-25T07:19:35.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:35.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2024-04-25T07:19:36.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:36.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2024-04-25T07:19:36.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:36.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2024-04-25T07:19:36.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:36.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2024-04-25T07:19:36.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:36.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2024-04-25T07:19:37.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:37.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2024-04-25T07:19:37.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:37.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2024-04-25T07:19:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2024-04-25T07:19:37.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:37.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2024-04-25T07:19:38.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:38.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2024-04-25T07:19:38.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:38.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2024-04-25T07:19:38.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:38.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2024-04-25T07:19:38.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:38.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2024-04-25T07:19:39.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:39.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2024-04-25T07:19:39.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:39.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2024-04-25T07:19:39.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:39.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2024-04-25T07:19:39.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:39.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2024-04-25T07:19:39.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:39.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2024-04-25T07:19:40.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:40.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2024-04-25T07:19:40.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:40.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2024-04-25T07:19:40.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:40.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2024-04-25T07:19:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:40.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2024-04-25T07:19:40.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:40.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2024-04-25T07:19:41.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:41.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2024-04-25T07:19:41.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:41.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2024-04-25T07:19:41.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:41.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2024-04-25T07:19:41.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:41.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2024-04-25T07:19:42.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:42.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2024-04-25T07:19:42.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:42.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2024-04-25T07:19:42.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:42.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2024-04-25T07:19:42.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:42.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2024-04-25T07:19:42.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:42.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2024-04-25T07:19:42.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:42.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2024-04-25T07:19:43.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:43.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2024-04-25T07:19:43.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:43.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2024-04-25T07:19:43.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:43.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2024-04-25T07:19:43.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:43.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2024-04-25T07:19:43.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:43.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2024-04-25T07:19:44.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:44.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2024-04-25T07:19:44.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:44.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2024-04-25T07:19:44.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:44.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2024-04-25T07:19:44.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:44.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2024-04-25T07:19:44.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:44.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2024-04-25T07:19:45.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:45.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2024-04-25T07:19:45.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:45.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2024-04-25T07:19:45.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:45.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2024-04-25T07:19:45.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:45.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2024-04-25T07:19:45.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:45.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2024-04-25T07:19:46.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:46.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2024-04-25T07:19:46.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:46.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2024-04-25T07:19:46.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:46.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2024-04-25T07:19:46.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:46.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2024-04-25T07:19:46.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:46.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2024-04-25T07:19:47.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:47.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2024-04-25T07:19:47.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:47.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2024-04-25T07:19:47.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:47.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2024-04-25T07:19:47.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:47.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2024-04-25T07:19:47.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:47.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2024-04-25T07:19:47.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:686: TEST_divergent_3: for i in $(seq 1 $testobjects) 2024-04-25T07:19:47.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:688: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2024-04-25T07:19:48.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:19:48.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:691: TEST_divergent_3: wait_for_clean 2024-04-25T07:19:48.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:19:48.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:19:48.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:19:48.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:19:48.132 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:19:48.132 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:19:48.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:19:48.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:19:48.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:19:48.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:19:48.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:19:48.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:19:48.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:19:48.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:19:48.209 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:19:48.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:19:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:19:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:19:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:19:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:48.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:19:48.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-04-25T07:19:48.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-04-25T07:19:48.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-04-25T07:19:48.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:48.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:19:49.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672983 2024-04-25T07:19:49.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672983 2024-04-25T07:19:49.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672983' 2024-04-25T07:19:49.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:19:49.024 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:19:49.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509460 2024-04-25T07:19:49.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509460 2024-04-25T07:19:49.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672983 2-64424509460' 2024-04-25T07:19:49.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:49.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-04-25T07:19:49.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:49.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:19:49.184 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-04-25T07:19:49.184 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:49.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-04-25T07:19:49.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-04-25T07:19:49.186 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836516 2024-04-25T07:19:49.186 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:19:49.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836517 -lt 21474836516 2024-04-25T07:19:49.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:49.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672983 2024-04-25T07:19:49.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:49.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:19:49.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672983 2024-04-25T07:19:49.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:49.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672983 2024-04-25T07:19:49.595 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672983 2024-04-25T07:19:49.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-04-25T07:19:49.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:19:50.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672983 2024-04-25T07:19:50.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:19:50.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:19:50.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509460 2024-04-25T07:19:50.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:19:50.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509460 2024-04-25T07:19:50.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:19:50.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509460 2024-04-25T07:19:50.009 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509460 2024-04-25T07:19:50.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-04-25T07:19:50.010 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:19:50.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509460 -lt 64424509460 2024-04-25T07:19:50.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:19:50.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:50.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:51.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:19:51.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:19:51.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:19:51.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:19:51.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:19:51.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:19:51.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:19:51.010 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:19:51.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-04-25T07:19:51.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:19:51.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:19:51.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:19:52.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-04-25T07:19:52.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:19:52.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:19:52.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: get_pg test existing_1 2024-04-25T07:19:52.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:19:52.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:19:52.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:19:52.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:19:52.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:693: TEST_divergent_3: local pgid=1.1b 2024-04-25T07:19:52.417 INFO:tasks.workunit.client.0.smithi134.stdout:blackholing osds 0 2 2024-04-25T07:19:52.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: echo 'blackholing osds 0 2' 2024-04-25T07:19:52.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:697: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:19:52.823 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:35.258403+0000 36'7 36:64 [0,2,1] 0 [0,2,1] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:15:45.863951+0000 1 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:32.046982+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:35:32.323182+0000 1 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:35.032008+0000 36'8 36:60 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:52:33.910478+0000 0 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:33.104721+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:40:00.111576+0000 1 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:43.149836+0000 36'7 36:64 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:47:17.402767+0000 3 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 0 0 0 0 3085 0 0 8 0 8 active+clean 2024-04-25T07:19:44.150542+0000 36'8 36:65 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:06:51.946367+0000 3 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:46.140738+0000 36'7 36:64 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:18:48.031976+0000 2 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:42.219089+0000 36'6 36:63 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:33:02.087050+0000 2 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:38.235587+0000 36'8 36:60 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:39:28.875657+0000 1 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 0 0 0 3702 0 0 10 0 10 active+clean 2024-04-25T07:19:35.509885+0000 36'10 36:67 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:20:29.456072+0000 2 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:29.130462+0000 36'6 36:58 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:54:43.502131+0000 1 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 0 0 0 3085 0 0 9 0 9 active+clean 2024-04-25T07:19:32.613658+0000 36'9 36:66 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.707930+0000 2 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:19:33.675694+0000 36'5 36:51 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:19:28.707251+0000 0 0 2024-04-25T07:19:52.824 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:40.079673+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:10:17.427692+0000 2 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:34.501474+0000 36'6 36:52 [2,0,1] 2 [2,0,1] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:58:28.118158+0000 0 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:40.341410+0000 36'6 36:52 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:13:14.444486+0000 2 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:27.177413+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:11:40.637607+0000 0 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:38.460113+0000 36'7 36:63 [2,1,0] 2 [2,1,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:36:59.009150+0000 1 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 0 0 0 0 2468 0 0 7 0 7 active+clean 2024-04-25T07:19:45.317524+0000 36'7 36:64 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:31:53.444987+0000 3 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:33.676141+0000 36'8 36:65 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:00:36.492014+0000 2 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:34.281503+0000 36'7 36:53 [0,2,1] 0 [0,2,1] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:30:18.761066+0000 0 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 0 0 0 1851 0 0 6 0 6 active+clean 2024-04-25T07:19:36.287837+0000 36'6 36:52 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:39:24.988927+0000 1 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:19:39.267834+0000 36'5 36:51 [0,1,2] 0 [0,1,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:09:04.568728+0000 1 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:37.103436+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:27:58.945523+0000 1 0 2024-04-25T07:19:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:36.557200+0000 36'7 36:63 [2,0,1] 2 [2,0,1] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:02:15.899878+0000 1 0 2024-04-25T07:19:52.826 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:36.173496+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:47:25.803542+0000 2 0 2024-04-25T07:19:52.826 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:39.172529+0000 36'8 36:60 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:41:09.389502+0000 2 0 2024-04-25T07:19:52.826 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:19:38.218605+0000 34'5 36:61 [0,1,2] 0 [0,1,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:37:08.848680+0000 1 0 2024-04-25T07:19:52.826 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:37.455561+0000 36'7 36:53 [2,1,0] 2 [2,1,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:55:09.377937+0000 1 0 2024-04-25T07:19:52.826 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:34.113480+0000 36'6 36:58 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:22:08.449460+0000 1 0 2024-04-25T07:19:52.827 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:41.172463+0000 36'8 36:65 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:47:47.091531+0000 3 0 2024-04-25T07:19:52.827 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 0 0 0 3085 0 0 8 0 8 active+clean 2024-04-25T07:19:37.321801+0000 36'8 36:54 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:31:07.610460+0000 3 0 2024-04-25T07:19:52.827 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:19:52.827 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:19:52.827 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:19:52.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:52.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.0 2024-04-25T07:19:52.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:19:52.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:19:52.840 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:19:52.840 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:52.840 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:19:52.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.0.asok 2024-04-25T07:19:52.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:19:52.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:19:52.948 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:19:52.949 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:19:52.949 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:19:52.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:19:52.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: get_asok_path osd.2 2024-04-25T07:19:52.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:19:52.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:19:52.963 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:19:52.963 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:19:52.963 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:19:52.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.2.asok 2024-04-25T07:19:52.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:19:52.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:700: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:19:53.069 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:19:53.069 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:19:53.069 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:19:53.082 INFO:tasks.workunit.client.0.smithi134.stdout:create new divergent objects 2024-04-25T07:19:53.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: echo 'create new divergent objects' 2024-04-25T07:19:53.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: seq 1 2 2024-04-25T07:19:53.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:19:53.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:19:53.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_1 2024-04-25T07:19:53.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: rados -p test create newobject_2 2024-04-25T07:19:53.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:710: TEST_divergent_3: echo 'writing divergent objects' 2024-04-25T07:19:53.086 INFO:tasks.workunit.client.0.smithi134.stdout:writing divergent objects 2024-04-25T07:19:53.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: seq 1 2 2024-04-25T07:19:53.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:19:53.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2024-04-25T07:19:53.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2024-04-25T07:19:53.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: echo 'remove divergent objects' 2024-04-25T07:19:53.089 INFO:tasks.workunit.client.0.smithi134.stdout:remove divergent objects 2024-04-25T07:19:53.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2024-04-25T07:19:53.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: seq 1 2 2024-04-25T07:19:53.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:19:53.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 1 + 2 2024-04-25T07:19:53.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=3 2024-04-25T07:19:53.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:717: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2024-04-25T07:19:53.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_3 2024-04-25T07:19:53.094 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: expr 2 + 2 2024-04-25T07:19:53.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:719: TEST_divergent_3: rmi=4 2024-04-25T07:19:53.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:722: TEST_divergent_3: sleep 10 2024-04-25T07:19:53.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: rados -p test rm existing_4 2024-04-25T07:20:03.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:723: TEST_divergent_3: killall -9 rados 2024-04-25T07:20:03.103 INFO:tasks.workunit.client.0.smithi134.stderr:rados: no process found 2024-04-25T07:20:03.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: echo 'killing all the osds' 2024-04-25T07:20:03.103 INFO:tasks.workunit.client.0.smithi134.stdout:killing all the osds 2024-04-25T07:20:03.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:727: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:20:03.517 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:20:03.517 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:35.258403+0000 36'7 36:64 [0,2,1] 0 [0,2,1] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:15:45.863951+0000 1 0 2024-04-25T07:20:03.517 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:32.046982+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:35:32.323182+0000 1 0 2024-04-25T07:20:03.517 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:35.032008+0000 36'8 36:60 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:52:33.910478+0000 0 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:33.104721+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:40:00.111576+0000 1 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:43.149836+0000 36'7 36:64 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:47:17.402767+0000 3 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 0 0 0 0 3435 0 0 9 0 9 active+clean 2024-04-25T07:19:44.150542+0000 36'9 36:66 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:06:51.946367+0000 3 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:46.140738+0000 36'7 36:64 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:18:48.031976+0000 2 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:42.219089+0000 36'6 36:63 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:33:02.087050+0000 2 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:38.235587+0000 36'8 36:60 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:39:28.875657+0000 1 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 0 0 0 3702 0 0 10 0 10 active+clean 2024-04-25T07:19:35.509885+0000 36'10 36:67 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:20:29.456072+0000 2 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:29.130462+0000 36'6 36:58 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:54:43.502131+0000 1 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 0 0 0 3085 0 0 9 0 9 active+clean 2024-04-25T07:19:32.613658+0000 36'9 36:66 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.707930+0000 2 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:19:33.675694+0000 36'5 36:51 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:19:28.707251+0000 0 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:40.079673+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:10:17.427692+0000 2 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:34.501474+0000 36'6 36:52 [2,0,1] 2 [2,0,1] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:58:28.118158+0000 0 0 2024-04-25T07:20:03.518 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:40.341410+0000 36'6 36:52 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:13:14.444486+0000 2 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:27.177413+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:11:40.637607+0000 0 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:38.460113+0000 36'7 36:63 [2,1,0] 2 [2,1,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:36:59.009150+0000 1 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 0 0 0 0 1851 0 0 9 0 9 active+clean 2024-04-25T07:19:45.317524+0000 36'9 36:66 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:31:53.444987+0000 3 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:33.676141+0000 36'8 36:65 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:00:36.492014+0000 2 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:34.281503+0000 36'7 36:53 [0,2,1] 0 [0,2,1] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:30:18.761066+0000 0 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 2 0 0 0 0 1234 0 0 7 0 7 active+clean 2024-04-25T07:19:36.287837+0000 36'7 36:53 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:39:24.988927+0000 1 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:19:39.267834+0000 36'5 36:51 [0,1,2] 0 [0,1,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:09:04.568728+0000 1 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:37.103436+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:27:58.945523+0000 1 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:36.557200+0000 36'7 36:63 [2,0,1] 2 [2,0,1] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:02:15.899878+0000 1 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:36.173496+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:47:25.803542+0000 2 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:39.172529+0000 36'8 36:60 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:41:09.389502+0000 2 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:19:38.218605+0000 34'5 36:61 [0,1,2] 0 [0,1,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:37:08.848680+0000 1 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:19:37.455561+0000 36'7 36:53 [2,1,0] 2 [2,1,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:55:09.377937+0000 1 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:19:34.113480+0000 36'6 36:58 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:22:08.449460+0000 1 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:19:41.172463+0000 36'8 36:65 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:47:47.091531+0000 3 0 2024-04-25T07:20:03.519 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 6 0 0 0 0 3435 0 0 10 0 10 active+clean 2024-04-25T07:19:37.321801+0000 36'10 36:56 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:31:07.610460+0000 3 0 2024-04-25T07:20:03.520 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:20:03.520 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:20:03.520 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:20:03.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:20:03.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:20:03.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:20:03.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:20:03.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:20:03.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:20:03.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:20:03.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:20:03.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.0 2024-04-25T07:20:04.202 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:20:04.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:20:04.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.1 2024-04-25T07:20:04.963 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already down. 2024-04-25T07:20:04.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:729: TEST_divergent_3: for i in $osds 2024-04-25T07:20:04.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:731: TEST_divergent_3: ceph osd down osd.2 2024-04-25T07:20:05.567 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already down. 2024-04-25T07:20:05.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:20:05.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.0 2024-04-25T07:20:06.117 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T07:20:06.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:733: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:20:06.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: ceph osd out osd.2 2024-04-25T07:20:06.685 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already out. 2024-04-25T07:20:06.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2024-04-25T07:20:06.702 INFO:tasks.workunit.client.0.smithi134.stdout:bringing up non_divergent 0 2 2024-04-25T07:20:06.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:20:07.115 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:20:07.115 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:35.258403+0000 36'7 36:64 [0,2,1] 0 [0,2,1] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:15:45.863951+0000 1 0 2024-04-25T07:20:07.115 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:32.046982+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:35:32.323182+0000 1 0 2024-04-25T07:20:07.115 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 0 0 0 2468 0 0 8 0 8 stale+active+clean 2024-04-25T07:19:35.032008+0000 36'8 36:60 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:52:33.910478+0000 0 0 2024-04-25T07:20:07.115 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:33.104721+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:40:00.111576+0000 1 0 2024-04-25T07:20:07.115 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:43.149836+0000 36'7 36:64 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:47:17.402767+0000 3 0 2024-04-25T07:20:07.115 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 0 0 0 0 3435 0 0 9 0 9 stale+active+clean 2024-04-25T07:19:44.150542+0000 36'9 36:66 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:06:51.946367+0000 3 0 2024-04-25T07:20:07.115 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:46.140738+0000 36'7 36:64 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:18:48.031976+0000 2 0 2024-04-25T07:20:07.115 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 0 0 0 1234 0 0 6 0 6 stale+active+clean 2024-04-25T07:19:42.219089+0000 36'6 36:63 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:33:02.087050+0000 2 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 0 0 0 2468 0 0 8 0 8 stale+active+clean 2024-04-25T07:19:38.235587+0000 36'8 36:60 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:39:28.875657+0000 1 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 0 0 0 3702 0 0 10 0 10 stale+active+clean 2024-04-25T07:19:35.509885+0000 36'10 36:67 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:20:29.456072+0000 2 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 0 0 0 1234 0 0 6 0 6 stale+active+clean 2024-04-25T07:19:29.130462+0000 36'6 36:58 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:54:43.502131+0000 1 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 0 0 0 3085 0 0 9 0 9 stale+active+clean 2024-04-25T07:19:32.613658+0000 36'9 36:66 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:53:26.707930+0000 2 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 0 0 0 617 0 0 5 0 5 stale+active+clean 2024-04-25T07:19:33.675694+0000 36'5 36:51 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:19:28.707251+0000 0 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:40.079673+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:10:17.427692+0000 2 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 0 0 0 1234 0 0 6 0 6 stale+active+clean 2024-04-25T07:19:34.501474+0000 36'6 36:52 [2,0,1] 2 [2,0,1] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:58:28.118158+0000 0 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 0 0 0 1234 0 0 6 0 6 stale+active+clean 2024-04-25T07:19:40.341410+0000 36'6 36:52 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:13:14.444486+0000 2 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:27.177413+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:11:40.637607+0000 0 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:38.460113+0000 36'7 36:63 [2,1,0] 2 [2,1,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:36:59.009150+0000 1 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 0 0 0 0 1851 0 0 9 0 9 stale+active+clean 2024-04-25T07:19:45.317524+0000 36'9 36:66 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:31:53.444987+0000 3 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 0 0 0 2468 0 0 8 0 8 stale+active+clean 2024-04-25T07:19:33.676141+0000 36'8 36:65 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:00:36.492014+0000 2 0 2024-04-25T07:20:07.116 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:34.281503+0000 36'7 36:53 [0,2,1] 0 [0,2,1] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:30:18.761066+0000 0 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 2 0 0 0 0 1234 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:36.287837+0000 36'7 36:53 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:39:24.988927+0000 1 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 0 0 0 617 0 0 5 0 5 stale+active+clean 2024-04-25T07:19:39.267834+0000 36'5 36:51 [0,1,2] 0 [0,1,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:09:04.568728+0000 1 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:37.103436+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:27:58.945523+0000 1 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:36.557200+0000 36'7 36:63 [2,0,1] 2 [2,0,1] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:02:15.899878+0000 1 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:36.173496+0000 36'7 36:59 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:47:25.803542+0000 2 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 0 0 0 2468 0 0 8 0 8 stale+active+clean 2024-04-25T07:19:39.172529+0000 36'8 36:60 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:41:09.389502+0000 2 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 0 0 0 617 0 0 5 0 5 stale+active+clean 2024-04-25T07:19:38.218605+0000 34'5 36:61 [0,1,2] 0 [0,1,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:37:08.848680+0000 1 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 0 0 0 1851 0 0 7 0 7 stale+active+clean 2024-04-25T07:19:37.455561+0000 36'7 36:53 [2,1,0] 2 [2,1,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:55:09.377937+0000 1 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 0 0 0 1234 0 0 6 0 6 stale+active+clean 2024-04-25T07:19:34.113480+0000 36'6 36:58 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:22:08.449460+0000 1 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 0 0 0 2468 0 0 8 0 8 stale+active+clean 2024-04-25T07:19:41.172463+0000 36'8 36:65 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:47:47.091531+0000 3 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 6 0 0 0 0 3435 0 0 10 0 10 stale+active+clean 2024-04-25T07:19:37.321801+0000 36'10 36:56 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:31:07.610460+0000 3 0 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:20:07.117 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:20:07.118 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 0 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:20:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:20:07.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:20:07.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:20:07.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:20:07.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:20:07.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:20:07.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:20:07.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:20:07.133 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:20:07.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:20:07.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:20:07.135 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:20:07.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:20:07.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:20:07.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:20:07.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:20:07.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:20:07.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:20:07.167 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:07.170+0000 7fc5f6a827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:07.169 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:07.174+0000 7fc5f6a827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:07.170 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:07.174+0000 7fc5f6a827c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:07.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:20:07.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:20:07.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:20:07.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:20:07.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:20:07.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:07.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:20:07.535 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:20:07.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:07.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:20:07.702 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:07.706+0000 7fc5f6a827c0 -1 Falling back to public interface 2024-04-25T07:20:08.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:09.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:09.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:09.026 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:20:09.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:20:09.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:09.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:20:09.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:10.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:10.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:10.462 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:20:10.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:20:10.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:10.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:20:10.826 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:10.830+0000 7fc5f6a827c0 -1 osd.0 36 log_to_monitors true 2024-04-25T07:20:10.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:11.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:11.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:11.910 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:20:11.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:20:11.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:11.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:20:12.344 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up out weight 0 up_from 43 up_thru 35 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6802/3611123750,v1:127.0.0.1:6803/3611123750] [v2:127.0.0.1:6804/3611123750,v1:127.0.0.1:6805/3611123750] exists,up 690eaf5b-9dca-4c84-9e79-ba2565d401e0 2024-04-25T07:20:12.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:20:12.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:20:12.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:20:12.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:20:12.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: activate_osd td/divergent-priors 2 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:20:12.347 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:20:12.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:20:12.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:20:12.349 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:20:12.349 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:20:12.349 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:20:12.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:20:12.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:20:12.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:20:12.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:20:12.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:20:12.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:20:12.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:20:12.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:20:12.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:20:12.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:20:12.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:20:12.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:20:12.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:20:12.350 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:20:12.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:20:12.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:20:12.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:20:12.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:20:12.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:20:12.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:20:12.390 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:12.394+0000 7f67cdc0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:12.392 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:12.394+0000 7f67cdc0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:12.393 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:12.398+0000 7f67cdc0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:20:12.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:20:12.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:20:12.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:20:12.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:20:12.776 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:20:12.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:20:12.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:12.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:20:12.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:12.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:12.946 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:12.950+0000 7f67cdc0b7c0 -1 Falling back to public interface 2024-04-25T07:20:13.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:14.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:14.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:14.188 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:20:14.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:20:14.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:14.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:14.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:15.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:15.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:15.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:20:15.610 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:20:15.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:15.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:16.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:20:16.205 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:20:16.210+0000 7f67cdc0b7c0 -1 osd.2 36 log_to_monitors true 2024-04-25T07:20:17.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:20:17.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:20:17.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:20:17.191 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:20:17.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:20:17.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:20:17.665 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up out weight 0 up_from 45 up_thru 35 down_at 39 last_clean_interval [15,36) [v2:127.0.0.1:6810/492884850,v1:127.0.0.1:6811/492884850] [v2:127.0.0.1:6812/492884850,v1:127.0.0.1:6813/492884850] exists,up c427ab00-4be7-4143-8d54-955f5dac6f79 2024-04-25T07:20:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:20:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:20:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:20:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:20:17.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.0 2024-04-25T07:20:18.518 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already in. 2024-04-25T07:20:18.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:745: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:20:18.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph osd in osd.2 2024-04-25T07:20:19.690 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already in. 2024-04-25T07:20:19.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:20:19.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: wait_for_clean 2024-04-25T07:20:19.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:20:19.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:20:19.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:20:19.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:20:19.715 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:20:19.715 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:20:19.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:20:19.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:20:19.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:20:19.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:20:19.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:20:19.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:20:19.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:20:19.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:20:19.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:20:20.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:20:20.422 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:20:20.422 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:20:20.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:20:20.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:20.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:20:20.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=184683593732 2024-04-25T07:20:20.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 184683593732 2024-04-25T07:20:20.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-184683593732' 2024-04-25T07:20:20.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:20.601 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:20:20.725 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:20:20.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:20:20.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:20:20.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:20:20.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:20.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:20:20.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528323 2024-04-25T07:20:20.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528323 2024-04-25T07:20:20.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-184683593732 2-193273528323' 2024-04-25T07:20:20.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:20.974 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-184683593732 2024-04-25T07:20:20.974 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:20.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:20:20.977 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-184683593732 2024-04-25T07:20:20.977 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:20.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=184683593732 2024-04-25T07:20:20.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 184683593732' 2024-04-25T07:20:20.979 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 184683593732 2024-04-25T07:20:20.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:20:21.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 184683593732 -lt 184683593732 2024-04-25T07:20:21.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:21.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-193273528323 2024-04-25T07:20:21.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:21.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:20:21.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-193273528323 2024-04-25T07:20:21.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:21.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528323 2024-04-25T07:20:21.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 193273528323' 2024-04-25T07:20:21.441 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 193273528323 2024-04-25T07:20:21.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:20:21.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528323 -lt 193273528323 2024-04-25T07:20:21.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:20:21.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:21.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:22.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:20:22.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:22.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:22.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:22.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:22.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:22.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:22.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:22.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:22.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:22.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:22.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:23.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:23.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:20:23.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:20:23.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:20:23.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:23.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:20:23.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:23.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:23.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:23.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:23.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:23.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:23.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:23.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:23.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:23.968 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:23.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:23.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:24.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:24.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:24.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:24.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:24.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:24.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:24.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:24.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:24.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:25.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:25.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:25.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:20:25.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:25.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:20:25.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:25.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:25.297 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:25.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:25.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:25.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:25.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:25.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:25.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:25.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:25.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:25.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:26.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:26.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:26.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:26.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:26.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:26.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:26.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:26.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:26.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:26.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:26.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:26.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:20:26.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:26.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:20:27.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:27.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:27.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:27.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:27.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:27.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:27.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:27.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:27.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:27.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:27.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:27.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:28.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:28.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:28.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:28.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:28.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:28.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:28.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:28.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:28.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:28.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:28.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:28.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:20:28.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:28.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:20:29.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:29.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:29.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:29.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:29.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:29.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:29.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:29.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:30.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:30.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:30.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:30.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:30.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:30.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:30.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:30.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:30.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:30.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:30.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:30.576 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:30.576 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:31.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:31.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:31.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:20:31.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:31.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:20:32.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:32.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:32.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:32.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:32.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:32.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:32.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:32.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:33.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:33.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:33.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:33.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:33.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:33.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:33.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:33.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:33.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:33.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:33.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:33.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:33.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:34.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:34.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:34.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:20:34.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:34.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:20:37.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:37.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:37.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:37.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:37.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:37.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:37.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:37.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:37.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:37.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:37.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:37.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:38.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:38.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:39.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:39.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:39.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:20:39.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:39.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:20:45.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:45.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:45.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:45.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:45.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:45.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:45.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:45.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:45.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:45.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:45.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:45.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:46.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:46.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:46.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:46.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:46.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:46.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:46.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:46.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:46.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:47.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:47.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:47.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:20:47.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:20:47.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:20:54.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:20:54.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:20:54.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:20:54.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:20:54.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:20:54.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:20:54.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:20:54.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:20:54.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:20:54.778 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:20:54.779 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:20:54.779 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:20:55.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:20:55.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:20:55.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:20:55.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:20:55.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:20:55.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:20:55.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:20:55.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:20:55.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:20:55.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:20:55.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:20:55.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:20:55.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:20:56.395 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:20:56.395 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_fingerprint": "c00a649e-0b30-40d2-8013-5e7737e4fddf", 2024-04-25T07:20:56.395 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:20:56.395 INFO:tasks.workunit.client.0.smithi134.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:20:56.395 INFO:tasks.workunit.client.0.smithi134.stdout: "timestamp": "2024-04-25T07:20:56.401492+0000", 2024-04-25T07:20:56.395 INFO:tasks.workunit.client.0.smithi134.stdout: "tag": "", 2024-04-25T07:20:56.395 INFO:tasks.workunit.client.0.smithi134.stdout: "health": { 2024-04-25T07:20:56.395 INFO:tasks.workunit.client.0.smithi134.stdout: "status": "HEALTH_WARN", 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "checks": { 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 17 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [], 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "OSD_DOWN": { 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 osds down", 2024-04-25T07:20:56.396 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "osd.1 (root=default,host=smithi134) is down" 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: "PG_DEGRADED": { 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 32 2024-04-25T07:20:56.397 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.398 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.399 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.a is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.400 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.d is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.e is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.401 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.f is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.402 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.403 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [2,0]" 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.404 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:20:56.405 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: "mutes": [] 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_first_committed": 1, 2024-04-25T07:20:56.406 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_last_committed": 1, 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap": { 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:17:16.869988Z", 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:17:16.869988Z", 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release": 19, 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release_name": "squid", 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "election_strategy": 1, 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "disallowed_leaders: ": "", 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": false, 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "tiebreaker_mon": "", 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_ranks: ": "", 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "features": { 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "persistent": [ 2024-04-25T07:20:56.407 INFO:tasks.workunit.client.0.smithi134.stdout: "kraken", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "luminous", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "mimic", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap-prune", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "nautilus", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "octopus", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "pacific", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "elector-pinging", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "quincy", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "reef", 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "squid" 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "optional": [] 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "mons": [ 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.408 INFO:tasks.workunit.client.0.smithi134.stdout: "rank": 0, 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "a", 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 0 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 0, 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 0, 2024-04-25T07:20:56.409 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_location": "{}" 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "quorum": [ 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: 0 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap": { 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 52, 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:17:17.912487+0000", 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:20:20.173441+0000", 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "last_up_change": "2024-04-25T07:20:16.597677+0000", 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "last_in_change": "2024-04-25T07:20:19.051859+0000", 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:20:56.410 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_num": 5800384, 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_set": [ 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "nodown", 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "noin", 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "noout", 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "pglog_hardlimit", 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snapdirs", 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "recovery_deletes", 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "sortbitwise" 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_version": 9, 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_snapshot": "", 2024-04-25T07:20:56.411 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_max": 1, 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "max_osd": 3, 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "min_compat_client": "jewel", 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "require_osd_release": "squid", 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "allow_crimson": false, 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "pools": [ 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_name": "test", 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "create_time": "2024-04-25T07:19:10.955277+0000", 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": 1, 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_names": "hashpspool", 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 3, 2024-04-25T07:20:56.412 INFO:tasks.workunit.client.0.smithi134.stdout: "min_size": 2, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_rule": 0, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "object_hash": 2, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_autoscale_mode": "on", 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num": 32, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num": 32, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num_target": 32, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_target": 32, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_pending": 32, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pg_merge_meta": { 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "source_pgid": "0.0", 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "ready_epoch": 0, 2024-04-25T07:20:56.413 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_started": 0, 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": 0, 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "source_version": "0'0", 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "target_version": "0'0" 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "last_change": "35", 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend": "0", 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_preluminous": "33", 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "auid": 0, 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_seq": 0, 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_epoch": 0, 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_snaps": [], 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps": "[]", 2024-04-25T07:20:56.414 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_bytes": 0, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_objects": 0, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "tiers": [], 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "tier_of": -1, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "read_tier": -1, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "write_tier": -1, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_mode": "none", 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_bytes": 0, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_objects": 0, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_flush_age": 0, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_evict_age": 0, 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profile": "", 2024-04-25T07:20:56.415 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_params": { 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "none" 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_period": 0, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_count": 0, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "use_gmt_hitset": true, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "grade_table": [], 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "stripe_width": 0, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "expected_num_objects": 0, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "fast_read": false, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "options": {}, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "application_metadata": {}, 2024-04-25T07:20:56.416 INFO:tasks.workunit.client.0.smithi134.stdout: "read_balance": { 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "score_type": "Fair distribution", 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "score_acting": 1.7799999713897705, 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "score_stable": 1.7799999713897705, 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_acting": 1.190000057220459, 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_stable": 1.190000057220459, 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity": 1, 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "osds": [ 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.417 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "690eaf5b-9dca-4c84-9e79-ba2565d401e0", 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 5, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 36, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 43, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 49, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 37, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.418 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.419 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:20:56.420 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.421 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6803/3611123750", 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6805/3611123750", 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3611123750", 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3611123750", 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.422 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "8ebabd29-af92-4d13-bc6b-3f0ab12e5b80", 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 0, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 10, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 35, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 38, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:20:56.423 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.424 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.425 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.426 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/3928262223", 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/3928262223", 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3928262223", 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3928262223", 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout: "exists" 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "c427ab00-4be7-4143-8d54-955f5dac6f79", 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 15, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 36, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 45, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 51, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 39, 2024-04-25T07:20:56.428 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.429 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.430 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.431 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/492884850", 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/492884850", 2024-04-25T07:20:56.432 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/492884850", 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/492884850", 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_xinfo": [ 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:20:03.982995+0000", 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:20:56.433 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:17:52.962656+0000", 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:20:04.744871+0000", 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:18:28.893847+0000", 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.434 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:20:05.403591+0000", 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:19:06.690199+0000", 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap": [], 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_items": [], 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_primaries": [], 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_temp": [], 2024-04-25T07:20:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_temp": [], 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "blocklist": {}, 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "range_blocklist": {}, 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profiles": { 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "default": { 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "2", 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van" 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps_queue": [], 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "new_removed_snaps": [], 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "new_purged_snaps": [], 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_node_flags": {}, 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "device_class_flags": {}, 2024-04-25T07:20:56.436 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": { 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_enabled": false, 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_bucket_count": 0, 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_bucket": 0 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_metadata": [ 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6804/3611123750,v1:127.0.0.1:6805/3611123750]", 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:20:56.437 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:20:56.438 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:17:24.935678Z", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6802/3611123750,v1:127.0.0.1:6803/3611123750]", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:20:56.439 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3611123750,v1:127.0.0.1:6809/3611123750]", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3611123750,v1:127.0.0.1:6807/3611123750]", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.440 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/3928262223,v1:127.0.0.1:6813/3928262223]", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:20:56.441 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:18:00.704234Z", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:20:56.442 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/3928262223,v1:127.0.0.1:6811/3928262223]", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3928262223,v1:127.0.0.1:6817/3928262223]", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3928262223,v1:127.0.0.1:6815/3928262223]", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:20:56.443 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/492884850,v1:127.0.0.1:6813/492884850]", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:20:56.444 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:20:56.445 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:18:37.028861Z", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/492884850,v1:127.0.0.1:6811/492884850]", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/492884850,v1:127.0.0.1:6817/492884850]", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/492884850,v1:127.0.0.1:6815/492884850]", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:20:56.446 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_clean_epochs": { 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:20:56.447 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": { 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: "per_pool": [] 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_epochs": [ 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 43 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 45 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_first_committed": 1, 2024-04-25T07:20:56.448 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_last_committed": 52, 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: "crushmap": { 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": [ 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.0", 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.1", 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.449 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.2", 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: "types": [ 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 0, 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd" 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "host" 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.450 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 2, 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "chassis" 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 3, 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "rack" 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 4, 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "row" 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 5, 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pdu" 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.451 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 6, 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pod" 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 7, 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "room" 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 8, 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "datacenter" 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 9, 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "zone" 2024-04-25T07:20:56.452 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 10, 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "region" 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "root" 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: "buckets": [ 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -1, 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default", 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:20:56.453 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -2, 2024-04-25T07:20:56.454 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default~hdd", 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:20:56.455 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134", 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:20:56.456 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134~hdd", 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:20:56.457 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:20:56.458 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: "rules": [ 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 0, 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "replicated_rule", 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T07:20:56.459 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_firstn", 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: "tunables": { 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_tries": 0, 2024-04-25T07:20:56.460 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_total_tries": 50, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_stable": 1, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_descents": 100, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_collision_tries": 100, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "straw_calc_version": 1, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "profile": "jewel", 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_tunables": 1, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "legacy_tunables": 0, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "minimum_required_version": "jewel", 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables": 1, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables2": 1, 2024-04-25T07:20:56.461 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v2_rules": 0, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables3": 1, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v3_rules": 0, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v4_buckets": 1, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables5": 1, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v5_rules": 0, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "has_msr_rules": 0 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_args": {} 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "fsmap": { 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "default_fscid": -1, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": { 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": {}, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "ro_compat": {}, 2024-04-25T07:20:56.462 INFO:tasks.workunit.client.0.smithi134.stdout: "incompat": { 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_1": "base v0.20", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_8": "no anchor table", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_9": "file layout v2", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_11": "minor log segments", 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_flags": { 2024-04-25T07:20:56.463 INFO:tasks.workunit.client.0.smithi134.stdout: "enable_multiple": true, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "ever_enabled_multiple": true 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "standbys": [], 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "filesystems": [] 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "auth": { 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 5, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "num_secrets": 3 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 32, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_active": 32, 2024-04-25T07:20:56.464 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_unknown": 0, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osd": 3, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_sum": { 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 61854, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:20:56.465 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 59, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 59, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:20:56.466 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:20:56.467 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 0, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 0, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 224, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 224, 2024-04-25T07:20:56.468 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 64, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 64, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 0 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum": { 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 96, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83856, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2336, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:20:56.469 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314488944, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036678656, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2392064, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 552500, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:20:56.470 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum_by_class": { 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "hdd": { 2024-04-25T07:20:56.471 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 96, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83856, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2336, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314488944, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036678656, 2024-04-25T07:20:56.472 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2392064, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 552500, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:20:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_stats": [ 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T07:20:56.474 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 32, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 61854, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:20:56.475 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 59, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 59, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:20:56.476 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:20:56.477 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 1220608, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 184566, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 224, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 224, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 64, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 64, 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 3 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.478 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_stats": [ 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 184683593739 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 42949672986 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 193273528330 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.479 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_state": [ 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "active+undersized+degraded", 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 32 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_osd": [ 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 13, 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 32, 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:20:56.480 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 19, 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 32, 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 0, 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": [ 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": {} 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.481 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "servicemap": { 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 5, 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:20:07.271206+0000", 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "services": {} 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_last_committed": 98, 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_first_committed": 1, 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_last_committed": 422, 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "paxos": { 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 576, 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pn": 0, 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: "accepted_pn": 0 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:20:56.482 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:20:56.483 INFO:tasks.workunit.client.0.smithi134.stderr:report 3319582494 2024-04-25T07:20:56.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:20:56.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: expr 2 + 2 2024-04-25T07:20:56.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:753: TEST_divergent_3: objname=existing_4 2024-04-25T07:20:56.484 INFO:tasks.workunit.client.0.smithi134.stdout:writing non-divergent object existing_4 2024-04-25T07:20:56.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2024-04-25T07:20:56.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:20:56.856 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:20:56.856 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199129+0000 36'7 52:77 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:45.508012+0000 1 0 2024-04-25T07:20:56.856 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199437+0000 36'7 52:72 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:35:35.618696+0000 1 0 2024-04-25T07:20:56.856 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199272+0000 36'8 52:73 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:15:51.983403+0000 0 0 2024-04-25T07:20:56.856 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201597+0000 36'7 52:72 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:14:47.557165+0000 1 0 2024-04-25T07:20:56.856 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 3 0 0 1928 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.650880+0000 36'7 52:63 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:20:56.856 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.751530+0000 36'8 52:72 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:40:47.427359+0000 3 0 2024-04-25T07:20:56.856 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751387+0000 36'7 52:71 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:03.496960+0000 2 0 2024-04-25T07:20:56.856 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 2 0 0 1311 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.751394+0000 36'6 52:62 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.092015+0000 36'8 52:73 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:01:10.192117+0000 1 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 6 0 0 3702 0 0 10 0 10 active+undersized+degraded 2024-04-25T07:20:20.717983+0000 36'10 52:74 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:20:27.383293+0000 2 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200625+0000 36'6 52:71 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:49:11.655339+0000 1 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 5 0 0 3085 0 0 9 0 9 active+undersized+degraded 2024-04-25T07:20:20.752707+0000 36'9 52:73 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:35.949285+0000 2 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.091495+0000 36'5 52:64 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:40.789694+0000 0 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201028+0000 36'7 52:72 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:04:14.311200+0000 2 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.717818+0000 36'6 52:59 [2,0] 2 [2,0] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:04:56.967725+0000 0 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200403+0000 36'6 52:64 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:11:32.204428+0000 2 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.090395+0000 36'7 52:72 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:23:29.292482+0000 0 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.752303+0000 36'7 52:70 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:07:52.759656+0000 1 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 0 4 0 0 2468 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.752103+0000 36'7 52:71 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:19:51.893481+0000 3 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.718649+0000 36'8 52:72 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:02:34.054514+0000 2 0 2024-04-25T07:20:56.857 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.200430+0000 36'7 52:66 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:26:21.961190+0000 0 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 3 0 0 1851 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.198803+0000 36'6 52:65 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:32:12.875857+0000 1 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.201414+0000 36'5 52:63 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:52:53.913570+0000 1 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201995+0000 36'7 52:72 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:35:44.470339+0000 1 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.719193+0000 36'7 52:70 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:28.203431+0000 1 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.091040+0000 36'7 52:72 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:23.236032+0000 2 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.090445+0000 36'8 52:73 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:10:48.312816+0000 2 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.199383+0000 34'5 52:73 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:56:14.683494+0000 1 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751897+0000 36'7 52:60 [2,0] 2 [2,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:46:24.319275+0000 1 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.199845+0000 36'6 52:71 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:05:19.078412+0000 1 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.651015+0000 36'8 52:72 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:25.986079+0000 3 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199418+0000 36'8 52:67 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:19:22.411448+0000 3 0 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:20:56.858 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:20:56.859 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:20:56.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:20:56.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:20:56.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: wait_for_clean 2024-04-25T07:20:56.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:20:56.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:20:56.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:20:56.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:20:56.987 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:20:56.987 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:20:56.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:20:56.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:20:56.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:20:57.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:20:57.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:20:57.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:20:57.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:20:57.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:20:57.059 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:20:57.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:20:57.483 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:20:57.483 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:20:57.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:20:57.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:57.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:20:57.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=184683593742 2024-04-25T07:20:57.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 184683593742 2024-04-25T07:20:57.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-184683593742' 2024-04-25T07:20:57.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:57.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:20:57.766 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:20:57.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:20:57.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:20:57.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:20:57.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:20:57.768 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:20:57.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528333 2024-04-25T07:20:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528333 2024-04-25T07:20:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-184683593742 2-193273528333' 2024-04-25T07:20:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:57.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-184683593742 2024-04-25T07:20:57.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:57.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:20:57.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-184683593742 2024-04-25T07:20:57.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:57.946 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 184683593742 2024-04-25T07:20:57.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=184683593742 2024-04-25T07:20:57.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 184683593742' 2024-04-25T07:20:57.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:20:58.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 184683593740 -lt 184683593742 2024-04-25T07:20:58.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:20:59.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:20:59.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:20:59.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 184683593742 -lt 184683593742 2024-04-25T07:20:59.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:20:59.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-193273528333 2024-04-25T07:20:59.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:20:59.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:20:59.778 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-193273528333 2024-04-25T07:20:59.778 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:20:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528333 2024-04-25T07:20:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 193273528333' 2024-04-25T07:20:59.779 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 193273528333 2024-04-25T07:20:59.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:21:00.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528333 -lt 193273528333 2024-04-25T07:21:00.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:21:00.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:00.215 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:00.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-04-25T07:21:00.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:00.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:00.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:00.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:00.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:00.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:00.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:01.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:21:01.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:01.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:01.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:01.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:21:01.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:21:01.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:21:01.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:21:01.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:21:01.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:21:01.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:21:01.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:01.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:01.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:01.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:01.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:01.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:01.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:02.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:21:02.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:02.243 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:02.243 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:02.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:21:02.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:21:02.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:21:02.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:21:02.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:21:02.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:21:02.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:21:02.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:21:02.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:21:03.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:21:03.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:21:03.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:21:03.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:21:03.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:21:03.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:21:03.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:03.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:03.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:03.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:03.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:03.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:03.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:03.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:21:03.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:03.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:03.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:04.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:21:04.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:21:04.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:21:04.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:21:04.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:21:04.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:21:04.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:21:04.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:21:04.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:21:05.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:21:05.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:21:05.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:21:05.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:21:05.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:21:05.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:21:05.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:05.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:05.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:05.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:05.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:05.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:05.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:05.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:21:05.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:05.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:05.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:21:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:21:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:21:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:21:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:21:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:21:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:21:06.393 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:21:06.393 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:21:06.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:21:06.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:21:06.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:21:06.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:21:06.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:21:07.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:21:07.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:07.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:07.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:07.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:07.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:07.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:07.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:08.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:21:08.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:08.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:08.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:08.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:21:08.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:21:08.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:21:08.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:21:08.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:21:08.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:21:08.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:21:08.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:21:08.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:21:09.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:21:09.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:21:09.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:21:09.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:21:09.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:21:10.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:21:10.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:10.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:10.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:10.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:10.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:10.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:10.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:21:11.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:11.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:11.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:11.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:21:11.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:21:11.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:21:11.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:21:11.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:21:11.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:21:11.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:21:11.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:21:11.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:21:12.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:21:12.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:21:12.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:21:12.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:21:12.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:21:15.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:21:15.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:15.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:15.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:15.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:15.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:15.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:15.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:15.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:21:15.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:15.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:15.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:16.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:21:16.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:21:16.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:21:16.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:21:16.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:21:16.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:21:16.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:21:16.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:21:16.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:21:17.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:21:17.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:21:17.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:21:17.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:21:17.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:21:23.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:21:23.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:23.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:23.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:23.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:23.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:23.485 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:23.485 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:23.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:21:23.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:23.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:23.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:24.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:21:24.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:21:24.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:21:24.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:21:24.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:21:24.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:21:24.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:21:24.462 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:21:24.462 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:21:25.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:21:25.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:21:25.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:21:25.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:21:25.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:21:32.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:21:32.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:21:32.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:21:32.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:21:32.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:21:32.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:21:32.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:21:32.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:21:32.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:21:32.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:21:32.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:21:32.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:21:33.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 32 2024-04-25T07:21:33.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:21:33.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:21:33.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:21:33.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:21:33.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:21:33.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:21:33.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:21:33.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:21:33.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:21:33.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:21:33.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:21:33.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:21:34.481 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_fingerprint": "c00a649e-0b30-40d2-8013-5e7737e4fddf", 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "timestamp": "2024-04-25T07:21:34.489694+0000", 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "tag": "", 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "health": { 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "status": "HEALTH_WARN", 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "checks": { 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:21:34.482 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 17 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [], 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: "OSD_DOWN": { 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 osds down", 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.483 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "osd.1 (root=default,host=smithi134) is down" 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: "PG_DEGRADED": { 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 64 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.0 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.484 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.2 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.3 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.4 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.5 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.485 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.6 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.7 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.8 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.9 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.a is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.486 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.d is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.e is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.f is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.487 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.10 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.11 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.12 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.13 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.14 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.488 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.15 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.16 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.17 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.18 is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.19 is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.489 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1a is stuck undersized for 72s, current state active+undersized+degraded, last acting [2,0]" 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1b is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1c is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1d is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1e is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.490 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.1f is stuck undersized for 73s, current state active+undersized+degraded, last acting [0,2]" 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.491 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: "mutes": [] 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.492 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_first_committed": 1, 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_last_committed": 1, 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap": { 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:17:16.869988Z", 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:17:16.869988Z", 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release": 19, 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release_name": "squid", 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "election_strategy": 1, 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "disallowed_leaders: ": "", 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": false, 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "tiebreaker_mon": "", 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_ranks: ": "", 2024-04-25T07:21:34.493 INFO:tasks.workunit.client.0.smithi134.stdout: "features": { 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "persistent": [ 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "kraken", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "luminous", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "mimic", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap-prune", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "nautilus", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "octopus", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "pacific", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "elector-pinging", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "quincy", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "reef", 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: "squid" 2024-04-25T07:21:34.494 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.495 INFO:tasks.workunit.client.0.smithi134.stdout: "optional": [] 2024-04-25T07:21:34.495 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.495 INFO:tasks.workunit.client.0.smithi134.stdout: "mons": [ 2024-04-25T07:21:34.495 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.495 INFO:tasks.workunit.client.0.smithi134.stdout: "rank": 0, 2024-04-25T07:21:34.495 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "a", 2024-04-25T07:21:34.495 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 0 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:21:34.496 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 0, 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 0, 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_location": "{}" 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "quorum": [ 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: 0 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap": { 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 52, 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:17:17.912487+0000", 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:20:20.173441+0000", 2024-04-25T07:21:34.497 INFO:tasks.workunit.client.0.smithi134.stdout: "last_up_change": "2024-04-25T07:20:16.597677+0000", 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "last_in_change": "2024-04-25T07:20:19.051859+0000", 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_num": 5800384, 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_set": [ 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "nodown", 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "noin", 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "noout", 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "pglog_hardlimit", 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snapdirs", 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "recovery_deletes", 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: "sortbitwise" 2024-04-25T07:21:34.498 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_version": 9, 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_snapshot": "", 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_max": 1, 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "max_osd": 3, 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "min_compat_client": "jewel", 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "require_osd_release": "squid", 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "allow_crimson": false, 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "pools": [ 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.499 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_name": "test", 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "create_time": "2024-04-25T07:19:10.955277+0000", 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": 1, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_names": "hashpspool", 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 3, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "min_size": 2, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_rule": 0, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "object_hash": 2, 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_autoscale_mode": "on", 2024-04-25T07:21:34.500 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num": 32, 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num": 32, 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num_target": 32, 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_target": 32, 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_pending": 32, 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pg_merge_meta": { 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "source_pgid": "0.0", 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "ready_epoch": 0, 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_started": 0, 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": 0, 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "source_version": "0'0", 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "target_version": "0'0" 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "last_change": "35", 2024-04-25T07:21:34.501 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend": "0", 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_preluminous": "33", 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "auid": 0, 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_mode": "selfmanaged", 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_seq": 0, 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_epoch": 0, 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_snaps": [], 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps": "[]", 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_bytes": 0, 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_objects": 0, 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "tiers": [], 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "tier_of": -1, 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "read_tier": -1, 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "write_tier": -1, 2024-04-25T07:21:34.502 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_mode": "none", 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_bytes": 0, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_objects": 0, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_flush_age": 0, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_evict_age": 0, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profile": "", 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_params": { 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "none" 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_period": 0, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_count": 0, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "use_gmt_hitset": true, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:21:34.503 INFO:tasks.workunit.client.0.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "grade_table": [], 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "stripe_width": 0, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "expected_num_objects": 0, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "fast_read": false, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "options": {}, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "application_metadata": {}, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "read_balance": { 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "score_type": "Fair distribution", 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "score_acting": 1.7799999713897705, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "score_stable": 1.7799999713897705, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_score": 0.67000001668930054, 2024-04-25T07:21:34.504 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_acting": 1.190000057220459, 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_stable": 1.190000057220459, 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity_weighted": 1, 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity": 1, 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "osds": [ 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "690eaf5b-9dca-4c84-9e79-ba2565d401e0", 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:21:34.505 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 5, 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 36, 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 43, 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 49, 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 37, 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.506 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:21:34.507 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.508 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3611123750 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.509 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6803/3611123750", 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6805/3611123750", 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6809/3611123750", 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6807/3611123750", 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:21:34.510 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "8ebabd29-af92-4d13-bc6b-3f0ab12e5b80", 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 0, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 10, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 35, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 38, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:21:34.511 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.512 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:21:34.513 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:21:34.514 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3928262223 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/3928262223", 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/3928262223", 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3928262223", 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3928262223", 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: "exists" 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.515 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "c427ab00-4be7-4143-8d54-955f5dac6f79", 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 15, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 36, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 45, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 51, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 39, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.516 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.517 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.518 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:21:34.519 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 492884850 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/492884850", 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/492884850", 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/492884850", 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/492884850", 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.520 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_xinfo": [ 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:20:03.982995+0000", 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:17:52.962656+0000", 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:20:04.744871+0000", 2024-04-25T07:21:34.521 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:18:28.893847+0000", 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:20:05.403591+0000", 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:19:06.690199+0000", 2024-04-25T07:21:34.522 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap": [], 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_items": [], 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_primaries": [], 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_temp": [], 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_temp": [], 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "blocklist": {}, 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "range_blocklist": {}, 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profiles": { 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "default": { 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "2", 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T07:21:34.523 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van" 2024-04-25T07:21:34.525 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.525 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.525 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps_queue": [], 2024-04-25T07:21:34.525 INFO:tasks.workunit.client.0.smithi134.stdout: "new_removed_snaps": [], 2024-04-25T07:21:34.525 INFO:tasks.workunit.client.0.smithi134.stdout: "new_purged_snaps": [], 2024-04-25T07:21:34.525 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_node_flags": {}, 2024-04-25T07:21:34.525 INFO:tasks.workunit.client.0.smithi134.stdout: "device_class_flags": {}, 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": { 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_enabled": false, 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_bucket_count": 0, 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_bucket": 0 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_metadata": [ 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6804/3611123750,v1:127.0.0.1:6805/3611123750]", 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:21:34.526 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:21:34.527 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:17:24.935678Z", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6802/3611123750,v1:127.0.0.1:6803/3611123750]", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/3611123750,v1:127.0.0.1:6809/3611123750]", 2024-04-25T07:21:34.528 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/3611123750,v1:127.0.0.1:6807/3611123750]", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:21:34.529 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/3928262223,v1:127.0.0.1:6813/3928262223]", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:21:34.530 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:18:00.704234Z", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:21:34.531 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/3928262223,v1:127.0.0.1:6811/3928262223]", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3928262223,v1:127.0.0.1:6817/3928262223]", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3928262223,v1:127.0.0.1:6815/3928262223]", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:21:34.532 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/492884850,v1:127.0.0.1:6813/492884850]", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:21:34.533 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:21:34.534 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:18:37.028861Z", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/492884850,v1:127.0.0.1:6811/492884850]", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/492884850,v1:127.0.0.1:6817/492884850]", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/492884850,v1:127.0.0.1:6815/492884850]", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:21:34.535 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_clean_epochs": { 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": { 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: "per_pool": [] 2024-04-25T07:21:34.536 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_epochs": [ 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 43 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 45 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_first_committed": 1, 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_last_committed": 52, 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "crushmap": { 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": [ 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.537 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:21:34.538 INFO:tasks.workunit.client.0.smithi134.stderr:report 3086030278 2024-04-25T07:21:34.538 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.0", 2024-04-25T07:21:34.538 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:21:34.538 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.538 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.538 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.1", 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.2", 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "types": [ 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 0, 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd" 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:21:34.539 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "host" 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 2, 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "chassis" 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 3, 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "rack" 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 4, 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "row" 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 5, 2024-04-25T07:21:34.540 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pdu" 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 6, 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pod" 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 7, 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "room" 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 8, 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "datacenter" 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.541 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 9, 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "zone" 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 10, 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "region" 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "root" 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: "buckets": [ 2024-04-25T07:21:34.542 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -1, 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default", 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.543 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -2, 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default~hdd", 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.544 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134", 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:21:34.545 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134~hdd", 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:21:34.546 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:21:34.547 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: "rules": [ 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 0, 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "replicated_rule", 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T07:21:34.548 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_firstn", 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: "tunables": { 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_tries": 0, 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_total_tries": 50, 2024-04-25T07:21:34.549 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_stable": 1, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_descents": 100, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_collision_tries": 100, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "straw_calc_version": 1, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "profile": "jewel", 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_tunables": 1, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "legacy_tunables": 0, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "minimum_required_version": "jewel", 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables": 1, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables2": 1, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v2_rules": 0, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables3": 1, 2024-04-25T07:21:34.550 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v3_rules": 0, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v4_buckets": 1, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables5": 1, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v5_rules": 0, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "has_msr_rules": 0 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_args": {} 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "fsmap": { 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "default_fscid": -1, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": { 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": {}, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "ro_compat": {}, 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "incompat": { 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_1": "base v0.20", 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:21:34.551 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_8": "no anchor table", 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_9": "file layout v2", 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_11": "minor log segments", 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_flags": { 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "enable_multiple": true, 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "ever_enabled_multiple": true 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "standbys": [], 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: "filesystems": [] 2024-04-25T07:21:34.552 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "auth": { 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 5, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "num_secrets": 3 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 32, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_active": 32, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_unknown": 0, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osd": 3, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_sum": { 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 62204, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:21:34.553 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 60, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 60, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:21:34.554 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:21:34.555 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:21:34.556 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 225, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 225, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 64, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 64, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 0 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum": { 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 96, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:21:34.557 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83856, 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2336, 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314488944, 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036678656, 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:21:34.558 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2392064, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 553200, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:21:34.559 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum_by_class": { 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "hdd": { 2024-04-25T07:21:34.560 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:21:34.561 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:21:34.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 96, 2024-04-25T07:21:34.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:21:34.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:21:34.561 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:21:34.561 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:21:34.561 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83856, 2024-04-25T07:21:34.561 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2336, 2024-04-25T07:21:34.561 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314488944, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036678656, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2392064, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 553200, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:21:34.562 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:21:34.563 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_stats": [ 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 32, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 62204, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 100, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 300, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 100, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:21:34.564 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 100, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 60, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 60, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:21:34.565 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:21:34.566 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 1220608, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 185266, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:21:34.567 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 225, 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 225, 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 64, 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 64, 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 3 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_stats": [ 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 184683593749 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.568 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 42949672986 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 193273528340 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_state": [ 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "active+undersized+degraded", 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 32 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_osd": [ 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.569 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 13, 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 32, 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 19, 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 32, 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:21:34.570 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 0, 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": [ 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:21:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": {} 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "servicemap": { 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 7, 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:21:19.297656+0000", 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "services": {} 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_last_committed": 117, 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_first_committed": 1, 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_last_committed": 490, 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "paxos": { 2024-04-25T07:21:34.572 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:21:34.573 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 663, 2024-04-25T07:21:34.573 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pn": 0, 2024-04-25T07:21:34.573 INFO:tasks.workunit.client.0.smithi134.stdout: "accepted_pn": 0 2024-04-25T07:21:34.573 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:21:34.573 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:21:34.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:21:34.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: echo 'delay recovery' 2024-04-25T07:21:34.573 INFO:tasks.workunit.client.0.smithi134.stdout:delay recovery 2024-04-25T07:21:34.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:34.900 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:34.900 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199129+0000 36'7 52:77 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:45.508012+0000 1 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199437+0000 36'7 52:72 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:35:35.618696+0000 1 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199272+0000 36'8 52:73 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:15:51.983403+0000 0 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201597+0000 36'7 52:72 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:14:47.557165+0000 1 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 3 0 0 1928 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.650880+0000 36'7 52:63 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.751530+0000 36'8 52:72 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:40:47.427359+0000 3 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751387+0000 36'7 52:71 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:03.496960+0000 2 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 2 0 0 1311 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.751394+0000 36'6 52:62 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.092015+0000 36'8 52:73 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:01:10.192117+0000 1 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 6 0 0 3702 0 0 10 0 10 active+undersized+degraded 2024-04-25T07:20:20.717983+0000 36'10 52:74 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:20:27.383293+0000 2 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200625+0000 36'6 52:71 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:49:11.655339+0000 1 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 5 0 0 3085 0 0 9 0 9 active+undersized+degraded 2024-04-25T07:20:20.752707+0000 36'9 52:73 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:35.949285+0000 2 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.091495+0000 36'5 52:64 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:40.789694+0000 0 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201028+0000 36'7 52:72 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:04:14.311200+0000 2 0 2024-04-25T07:21:34.901 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.717818+0000 36'6 52:59 [2,0] 2 [2,0] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:04:56.967725+0000 0 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200403+0000 36'6 52:64 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:11:32.204428+0000 2 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.090395+0000 36'7 52:72 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:23:29.292482+0000 0 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.752303+0000 36'7 52:70 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:07:52.759656+0000 1 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 0 4 0 0 2818 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.752103+0000 52'8 52:72 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:19:51.893481+0000 3 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.718649+0000 36'8 52:72 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:02:34.054514+0000 2 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.200430+0000 36'7 52:66 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:26:21.961190+0000 0 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 3 0 0 1851 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.198803+0000 36'6 52:65 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:32:12.875857+0000 1 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.201414+0000 36'5 52:63 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:52:53.913570+0000 1 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201995+0000 36'7 52:72 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:35:44.470339+0000 1 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.719193+0000 36'7 52:70 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:28.203431+0000 1 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.091040+0000 36'7 52:72 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:23.236032+0000 2 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.090445+0000 36'8 52:73 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:10:48.312816+0000 2 0 2024-04-25T07:21:34.902 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.199383+0000 34'5 52:73 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:56:14.683494+0000 1 0 2024-04-25T07:21:34.903 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751897+0000 36'7 52:60 [2,0] 2 [2,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:46:24.319275+0000 1 0 2024-04-25T07:21:34.903 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.199845+0000 36'6 52:71 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:05:19.078412+0000 1 0 2024-04-25T07:21:34.903 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.651015+0000 36'8 52:72 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:25.986079+0000 3 0 2024-04-25T07:21:34.903 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199418+0000 36'8 52:67 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:19:22.411448+0000 3 0 2024-04-25T07:21:34.903 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:21:34.903 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:34.903 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:21:34.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:21:34.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.0 2024-04-25T07:21:34.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:21:34.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:21:34.917 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:34.917 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:34.917 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:21:34.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.0.asok 2024-04-25T07:21:34.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:21:34.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:21:35.015 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:21:35.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:763: TEST_divergent_3: for i in $non_divergent 2024-04-25T07:21:35.027 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: get_asok_path osd.2 2024-04-25T07:21:35.027 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:21:35.027 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:21:35.028 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:35.028 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:35.028 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:21:35.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.2.asok 2024-04-25T07:21:35.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:21:35.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:765: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:21:35.127 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:21:35.139 INFO:tasks.workunit.client.0.smithi134.stdout:revive divergent 1 2024-04-25T07:21:35.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:769: TEST_divergent_3: echo 'revive divergent 1' 2024-04-25T07:21:35.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:35.542 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:35.545 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199129+0000 36'7 52:77 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:45.508012+0000 1 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199437+0000 36'7 52:72 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:35:35.618696+0000 1 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199272+0000 36'8 52:73 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:15:51.983403+0000 0 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201597+0000 36'7 52:72 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:14:47.557165+0000 1 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 3 0 0 1928 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.650880+0000 36'7 52:63 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.751530+0000 36'8 52:72 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:40:47.427359+0000 3 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751387+0000 36'7 52:71 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:03.496960+0000 2 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 2 0 0 1311 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.751394+0000 36'6 52:62 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.092015+0000 36'8 52:73 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:01:10.192117+0000 1 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 6 0 0 3702 0 0 10 0 10 active+undersized+degraded 2024-04-25T07:20:20.717983+0000 36'10 52:74 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:20:27.383293+0000 2 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200625+0000 36'6 52:71 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:49:11.655339+0000 1 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 5 0 0 3085 0 0 9 0 9 active+undersized+degraded 2024-04-25T07:20:20.752707+0000 36'9 52:73 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:35.949285+0000 2 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.091495+0000 36'5 52:64 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:40.789694+0000 0 0 2024-04-25T07:21:35.546 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201028+0000 36'7 52:72 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:04:14.311200+0000 2 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.717818+0000 36'6 52:59 [2,0] 2 [2,0] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:04:56.967725+0000 0 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200403+0000 36'6 52:64 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:11:32.204428+0000 2 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.090395+0000 36'7 52:72 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:23:29.292482+0000 0 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.752303+0000 36'7 52:70 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:07:52.759656+0000 1 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 0 4 0 0 2818 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.752103+0000 52'8 52:72 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:19:51.893481+0000 3 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.718649+0000 36'8 52:72 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:02:34.054514+0000 2 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.200430+0000 36'7 52:66 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:26:21.961190+0000 0 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 3 0 0 1851 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.198803+0000 36'6 52:65 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:32:12.875857+0000 1 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.201414+0000 36'5 52:63 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:52:53.913570+0000 1 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201995+0000 36'7 52:72 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:35:44.470339+0000 1 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.719193+0000 36'7 52:70 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:28.203431+0000 1 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.091040+0000 36'7 52:72 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:23.236032+0000 2 0 2024-04-25T07:21:35.547 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.090445+0000 36'8 52:73 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:10:48.312816+0000 2 0 2024-04-25T07:21:35.548 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.199383+0000 34'5 52:73 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:56:14.683494+0000 1 0 2024-04-25T07:21:35.548 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751897+0000 36'7 52:60 [2,0] 2 [2,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:46:24.319275+0000 1 0 2024-04-25T07:21:35.548 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.199845+0000 36'6 52:71 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:05:19.078412+0000 1 0 2024-04-25T07:21:35.548 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.651015+0000 36'8 52:72 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:25.986079+0000 3 0 2024-04-25T07:21:35.548 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199418+0000 36'8 52:67 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:19:22.411448+0000 3 0 2024-04-25T07:21:35.548 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:21:35.548 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:35.548 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:21:35.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: ceph osd set noup 2024-04-25T07:21:36.200 INFO:tasks.workunit.client.0.smithi134.stderr:noup is set 2024-04-25T07:21:36.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-04-25T07:21:36.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:21:36.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:21:36.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:21:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:21:36.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:21:36.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:21:36.226 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:21:36.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:21:36.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:21:36.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:21:36.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:21:36.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:21:36.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:21:36.266 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:36.273+0000 7f2e7e76b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:36.310 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:36.317+0000 7f2e7e76b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:36.311 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:36.321+0000 7f2e7e76b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:36.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:21:36.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: sleep 5 2024-04-25T07:21:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:36.853+0000 7f2e7e76b7c0 -1 Falling back to public interface 2024-04-25T07:21:39.929 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:39.937+0000 7f2e7e76b7c0 -1 osd.1 36 log_to_monitors true 2024-04-25T07:21:41.696 INFO:tasks.workunit.client.0.smithi134.stdout:delay recovery divergent 2024-04-25T07:21:41.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:775: TEST_divergent_3: echo 'delay recovery divergent' 2024-04-25T07:21:41.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:776: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:42.121 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:42.121 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199129+0000 36'7 53:79 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:45.508012+0000 1 0 2024-04-25T07:21:42.121 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199437+0000 36'7 53:74 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:35:35.618696+0000 1 0 2024-04-25T07:21:42.121 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199272+0000 36'8 53:75 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:15:51.983403+0000 0 0 2024-04-25T07:21:42.121 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201597+0000 36'7 53:74 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:14:47.557165+0000 1 0 2024-04-25T07:21:42.121 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 3 0 0 1928 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.650880+0000 36'7 54:67 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:21:40.012147+0000 0 0 2024-04-25T07:21:42.121 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.751530+0000 36'8 54:76 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:40:47.427359+0000 3 0 2024-04-25T07:21:42.121 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751387+0000 36'7 54:75 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:03.496960+0000 2 0 2024-04-25T07:21:42.121 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 2 0 0 1311 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.751394+0000 36'6 54:66 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:21:40.012170+0000 0 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.092015+0000 36'8 53:75 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:01:10.192117+0000 1 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 6 0 0 3702 0 0 10 0 10 active+undersized+degraded 2024-04-25T07:20:20.717983+0000 36'10 54:78 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:20:27.383293+0000 2 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200625+0000 36'6 53:73 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:49:11.655339+0000 1 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 5 0 0 3085 0 0 9 0 9 active+undersized+degraded 2024-04-25T07:20:20.752707+0000 36'9 54:77 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:35.949285+0000 2 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.091495+0000 36'5 53:66 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:40.789694+0000 0 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201028+0000 36'7 53:74 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:04:14.311200+0000 2 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.717818+0000 36'6 54:63 [2,0] 2 [2,0] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:04:56.967725+0000 0 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200403+0000 36'6 53:66 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:11:32.204428+0000 2 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.090395+0000 36'7 53:74 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:23:29.292482+0000 0 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.752303+0000 36'7 54:74 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:07:52.759656+0000 1 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 0 4 0 0 2818 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.752103+0000 52'8 54:76 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:19:51.893481+0000 3 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.718649+0000 36'8 54:76 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:02:34.054514+0000 2 0 2024-04-25T07:21:42.122 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.200430+0000 36'7 53:68 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:26:21.961190+0000 0 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 3 0 0 1851 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.198803+0000 36'6 53:67 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:32:12.875857+0000 1 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.201414+0000 36'5 53:65 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:52:53.913570+0000 1 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201995+0000 36'7 53:74 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:35:44.470339+0000 1 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.719193+0000 36'7 54:74 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:28.203431+0000 1 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.091040+0000 36'7 53:74 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:23.236032+0000 2 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.090445+0000 36'8 53:75 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:10:48.312816+0000 2 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.199383+0000 34'5 53:75 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:56:14.683494+0000 1 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751897+0000 36'7 54:64 [2,0] 2 [2,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:46:24.319275+0000 1 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.199845+0000 36'6 53:73 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:05:19.078412+0000 1 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.651015+0000 36'8 54:76 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:25.986079+0000 3 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199418+0000 36'8 53:69 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:19:22.411448+0000 3 0 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:21:42.123 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:42.124 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:21:42.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: get_asok_path osd.1 2024-04-25T07:21:42.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:21:42.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:21:42.138 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:21:42.138 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:42.138 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:21:42.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.1.asok 2024-04-25T07:21:42.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:21:42.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:21:42.236 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:21:42.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: ceph osd unset noup 2024-04-25T07:21:42.897 INFO:tasks.workunit.client.0.smithi134.stderr:noup is unset 2024-04-25T07:21:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:781: TEST_divergent_3: wait_for_osd up 0 2024-04-25T07:21:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:21:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:42.921 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:21:42.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:42.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:21:43.781 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 43 up_thru 49 down_at 37 last_clean_interval [5,36) [v2:127.0.0.1:6802/3611123750,v1:127.0.0.1:6803/3611123750] [v2:127.0.0.1:6804/3611123750,v1:127.0.0.1:6805/3611123750] exists,up 690eaf5b-9dca-4c84-9e79-ba2565d401e0 2024-04-25T07:21:43.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:43.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:43.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:43.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: wait_for_osd up 1 2024-04-25T07:21:43.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:43.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:21:43.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:43.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:43.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:43.783 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:21:43.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:43.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:43.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:21:44.306 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 38 last_clean_interval [10,36) [v2:127.0.0.1:6818/1546537953,v1:127.0.0.1:6819/1546537953] [v2:127.0.0.1:6820/1546537953,v1:127.0.0.1:6821/1546537953] exists,up 8ebabd29-af92-4d13-bc6b-3f0ab12e5b80 2024-04-25T07:21:44.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:44.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:44.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:44.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: wait_for_osd up 2 2024-04-25T07:21:44.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:44.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:21:44.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:44.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:44.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:44.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:44.308 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:21:44.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:44.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:21:44.777 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 45 up_thru 57 down_at 39 last_clean_interval [15,36) [v2:127.0.0.1:6810/492884850,v1:127.0.0.1:6811/492884850] [v2:127.0.0.1:6812/492884850,v1:127.0.0.1:6813/492884850] exists,up c427ab00-4be7-4143-8d54-955f5dac6f79 2024-04-25T07:21:44.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:44.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:44.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:44.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:785: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:45.178 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:45.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199129+0000 36'7 55:83 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:45.508012+0000 1 0 2024-04-25T07:21:45.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199437+0000 36'7 55:78 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:35:35.618696+0000 1 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199272+0000 36'8 55:79 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:15:51.983403+0000 0 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201597+0000 36'7 55:78 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:14:47.557165+0000 1 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 3 0 0 1928 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.650880+0000 36'7 55:69 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.751530+0000 36'8 55:78 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:40:47.427359+0000 3 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751387+0000 36'7 55:77 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:30:03.496960+0000 2 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 2 0 0 1311 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.751394+0000 36'6 55:68 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.092015+0000 36'8 55:79 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:01:10.192117+0000 1 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 6 0 0 3702 0 0 10 0 10 active+undersized+degraded 2024-04-25T07:20:20.717983+0000 36'10 55:80 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:20:27.383293+0000 2 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200625+0000 36'6 55:77 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:49:11.655339+0000 1 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 5 0 0 3085 0 0 9 0 9 active+undersized+degraded 2024-04-25T07:20:20.752707+0000 36'9 55:79 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:20:35.949285+0000 2 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.091495+0000 36'5 55:70 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:40.789694+0000 0 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201028+0000 36'7 55:78 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:04:14.311200+0000 2 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.717818+0000 36'6 55:65 [2,0] 2 [2,0] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:04:56.967725+0000 0 0 2024-04-25T07:21:45.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200403+0000 36'6 55:70 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:11:32.204428+0000 2 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.090395+0000 36'7 55:78 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:23:29.292482+0000 0 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.752303+0000 36'7 55:76 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:07:52.759656+0000 1 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 0 4 0 0 2818 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.752103+0000 52'8 55:78 [2,0] 2 [2,0] 2 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:19:51.893481+0000 3 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.718649+0000 36'8 55:78 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:02:34.054514+0000 2 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.200430+0000 36'7 55:72 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:26:21.961190+0000 0 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 3 0 0 1851 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.198803+0000 36'6 55:71 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:32:12.875857+0000 1 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.201414+0000 36'5 55:69 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:52:53.913570+0000 1 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.201995+0000 36'7 55:78 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:35:44.470339+0000 1 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.719193+0000 36'7 55:76 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:28.203431+0000 1 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.091040+0000 36'7 55:78 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:47:23.236032+0000 2 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.090445+0000 36'8 55:79 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:10:48.312816+0000 2 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.199383+0000 34'5 55:79 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:56:14.683494+0000 1 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751897+0000 36'7 55:66 [2,0] 2 [2,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:46:24.319275+0000 1 0 2024-04-25T07:21:45.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.199845+0000 36'6 55:77 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:05:19.078412+0000 1 0 2024-04-25T07:21:45.181 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.651015+0000 36'8 55:78 [2,0] 2 [2,0] 2 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:01:25.986079+0000 3 0 2024-04-25T07:21:45.181 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199418+0000 36'8 55:73 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:19:22.411448+0000 3 0 2024-04-25T07:21:45.181 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:21:45.181 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:45.181 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:21:45.193 INFO:tasks.workunit.client.0.smithi134.stdout:wait for peering 2024-04-25T07:21:45.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'wait for peering' 2024-04-25T07:21:45.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199129+0000 36'7 55:83 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:45.508012+0000 1 0 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865318+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:42:44.993757+0000 1 0 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.875664+0000 36'8 59:74 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:00:25.396252+0000 0 0 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.874256+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:44:27.649616+0000 1 0 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 0 0 0 1928 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865937+0000 36'7 59:77 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:21:48.902246+0000 0 0 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 1 1 0 0 3085 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:43.877188+0000 36'8 59:78 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865174+0000 36'7 59:78 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:49:03.256342+0000 2 0 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 0 0 0 1311 0 0 6 0 6 active+clean 2024-04-25T07:21:43.867977+0000 36'6 59:76 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:21:48.902280+0000 0 0 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.873469+0000 36'8 59:74 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:28:36.113620+0000 1 0 2024-04-25T07:21:45.604 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 6 0 0 3702 0 0 10 0 10 active+undersized+degraded 2024-04-25T07:20:20.717983+0000 36'10 55:80 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:20:27.383293+0000 2 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:43.876190+0000 36'6 59:72 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:40:22.443601+0000 1 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 0 0 0 3085 0 0 9 0 9 active+clean 2024-04-25T07:21:43.863330+0000 36'9 59:80 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:09:58.082452+0000 2 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.091495+0000 36'5 55:70 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:40.789694+0000 0 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.867331+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:17:56.553200+0000 2 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.717818+0000 36'6 55:65 [2,0] 2 [2,0] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:04:56.967725+0000 0 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200403+0000 36'6 55:70 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:11:32.204428+0000 2 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.862708+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:25:49.991290+0000 0 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.752303+0000 36'7 55:76 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:07:52.759656+0000 1 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 1 1 0 0 2818 0 0 8 0 8 active+recovering+degraded 2024-04-25T07:21:43.878423+0000 52'8 59:78 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.718649+0000 36'8 55:78 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:02:34.054514+0000 2 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.200430+0000 36'7 55:72 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:26:21.961190+0000 0 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 3 0 0 1851 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.198803+0000 36'6 55:71 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:32:12.875857+0000 1 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.201414+0000 36'5 55:69 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:52:53.913570+0000 1 0 2024-04-25T07:21:45.605 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.876826+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:12:11.691007+0000 1 0 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.719193+0000 36'7 55:76 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:28.203431+0000 1 0 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865808+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:57:08.864139+0000 2 0 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.864381+0000 36'8 59:74 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:02.937326+0000 2 0 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.199383+0000 34'5 55:79 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:56:14.683494+0000 1 0 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751897+0000 36'7 55:66 [2,0] 2 [2,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:46:24.319275+0000 1 0 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:43.866635+0000 36'6 59:72 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:31:01.486204+0000 1 0 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.873709+0000 36'8 59:79 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:48:07.118691+0000 3 0 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199418+0000 36'8 55:73 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:19:22.411448+0000 3 0 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:45.606 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:21:45.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: rados -p test put foo /etc/fstab 2024-04-25T07:21:45.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: echo 'killing divergent 1' 2024-04-25T07:21:45.745 INFO:tasks.workunit.client.0.smithi134.stdout:killing divergent 1 2024-04-25T07:21:45.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:793: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:46.155 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:46.155 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.199129+0000 36'7 55:83 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:45.508012+0000 1 0 2024-04-25T07:21:46.155 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865318+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:42:44.993757+0000 1 0 2024-04-25T07:21:46.155 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.875664+0000 36'8 59:74 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:00:25.396252+0000 0 0 2024-04-25T07:21:46.155 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.874256+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:44:27.649616+0000 1 0 2024-04-25T07:21:46.155 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 0 0 0 1928 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865937+0000 36'7 59:77 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:21:48.902246+0000 0 0 2024-04-25T07:21:46.155 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 1 1 0 0 3085 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:43.877188+0000 36'8 59:78 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:21:46.155 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865174+0000 36'7 59:78 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:49:03.256342+0000 2 0 2024-04-25T07:21:46.155 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 0 0 0 1311 0 0 6 0 6 active+clean 2024-04-25T07:21:43.867977+0000 36'6 59:76 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:21:48.902280+0000 0 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.873469+0000 36'8 59:74 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:28:36.113620+0000 1 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 6 0 0 3702 0 0 10 0 10 active+undersized+degraded 2024-04-25T07:20:20.717983+0000 36'10 55:80 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:20:27.383293+0000 2 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:43.876190+0000 36'6 59:72 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:40:22.443601+0000 1 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 0 0 0 3085 0 0 9 0 9 active+clean 2024-04-25T07:21:43.863330+0000 36'9 59:80 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:09:58.082452+0000 2 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.091495+0000 36'5 55:70 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:40.789694+0000 0 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.867331+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:17:56.553200+0000 2 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.717818+0000 36'6 55:65 [2,0] 2 [2,0] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T19:04:56.967725+0000 0 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 2 0 0 1234 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.200403+0000 36'6 55:70 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:11:32.204428+0000 2 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.862708+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:25:49.991290+0000 0 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.752303+0000 36'7 55:76 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:07:52.759656+0000 1 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 1 1 0 0 2818 0 0 8 0 8 active+recovering+degraded 2024-04-25T07:21:43.878423+0000 52'8 59:78 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 4 0 0 2468 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.718649+0000 36'8 55:78 [2,0] 2 [2,0] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:02:34.054514+0000 2 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.200430+0000 36'7 55:72 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:26:21.961190+0000 0 0 2024-04-25T07:21:46.156 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 3 0 0 1851 0 0 6 0 6 active+undersized+degraded 2024-04-25T07:20:20.198803+0000 36'6 55:71 [0,2] 0 [0,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:32:12.875857+0000 1 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.201414+0000 36'5 55:69 [0,2] 0 [0,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:52:53.913570+0000 1 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.876826+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:12:11.691007+0000 1 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.719193+0000 36'7 55:76 [2,0] 2 [2,0] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:17:28.203431+0000 1 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865808+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:57:08.864139+0000 2 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.864381+0000 36'8 59:74 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:02.937326+0000 2 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 1 0 0 617 0 0 5 0 5 active+undersized+degraded 2024-04-25T07:20:20.199383+0000 34'5 55:79 [0,2] 0 [0,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:56:14.683494+0000 1 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 3 0 0 1851 0 0 7 0 7 active+undersized+degraded 2024-04-25T07:20:20.751897+0000 36'7 55:66 [2,0] 2 [2,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:46:24.319275+0000 1 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:43.866635+0000 36'6 59:72 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:31:01.486204+0000 1 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.873709+0000 36'8 59:79 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:48:07.118691+0000 3 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 5 0 0 3085 0 0 8 0 8 active+undersized+degraded 2024-04-25T07:20:20.199418+0000 36'8 55:73 [0,2] 0 [0,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:19:22.411448+0000 3 0 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:21:46.157 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:46.158 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:21:46.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:21:46.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:21:46.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:21:46.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:21:46.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:21:46.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:21:46.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:21:46.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: expfile=td/divergent-priors/exp.26105.out 2024-04-25T07:21:46.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:798: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.26105.out 2024-04-25T07:21:46.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:21:46.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:21:46.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:21:46.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:21:46.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:21:46.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.1b --file td/divergent-priors/exp.26105.out 2024-04-25T07:21:47.739 INFO:tasks.workunit.client.0.smithi134.stderr:Exporting 1.1b info 1.1b( v 36'8 lc 0'0 (0'0,36'8] local-lis/les=57/59 n=5 ec=33/23 lis/c=57/35 les/c/f=59/36/0 sis=57) 2024-04-25T07:21:47.740 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:21:47.747 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dcc4ff82:::existing_98:head# 2024-04-25T07:21:47.747 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd677bd3:::existing_35:head# 2024-04-25T07:21:47.754 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd6d90e7:::existing_46:head# 2024-04-25T07:21:47.755 INFO:tasks.workunit.client.0.smithi134.stderr:Export successful 2024-04-25T07:21:47.756 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T07:21:47.756 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T07:21:47.756 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.1b_head removing 1.1b 2024-04-25T07:21:47.889 INFO:tasks.workunit.client.0.smithi134.stderr:Remove successful 2024-04-25T07:21:48.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.26105.out 2024-04-25T07:21:48.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:21:48.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:21:48.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:21:48.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:21:48.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:21:48.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.26105.out 2024-04-25T07:21:49.221 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e59 pg_nums {1={23=1,33=32}} deleted_pools ) 2024-04-25T07:21:49.221 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.1b 2024-04-25T07:21:49.221 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d9cbd1d6:::existing_64:head# 2024-04-25T07:21:49.221 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:21:49.292 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dcc4ff82:::existing_98:head# 2024-04-25T07:21:49.292 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:21:49.326 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd677bd3:::existing_35:head# 2024-04-25T07:21:49.326 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:21:49.359 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd6d90e7:::existing_46:head# 2024-04-25T07:21:49.359 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T07:21:49.426 INFO:tasks.workunit.client.0.smithi134.stdout:write_pg epoch 59 info 1.1b( v 36'8 lc 0'0 (0'0,36'8] local-lis/les=57/59 n=5 ec=33/23 lis/c=57/35 les/c/f=59/36/0 sis=57) 2024-04-25T07:21:49.460 INFO:tasks.workunit.client.0.smithi134.stdout:Import successful 2024-04-25T07:21:49.998 INFO:tasks.workunit.client.0.smithi134.stdout:reviving divergent 1 2024-04-25T07:21:49.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:801: TEST_divergent_3: echo 'reviving divergent 1' 2024-04-25T07:21:49.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:21:50.399 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:21:50.399 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:44.333700+0000 36'7 59:96 [0,2,1] 0 [0,2,1] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:14:11.756780+0000 1 0 2024-04-25T07:21:50.399 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865318+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:42:44.993757+0000 1 0 2024-04-25T07:21:50.399 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.875664+0000 36'8 59:74 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:00:25.396252+0000 0 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.874256+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:44:27.649616+0000 1 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 0 0 0 1928 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865937+0000 36'7 59:77 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:21:48.902246+0000 0 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 1 1 0 0 3085 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:43.877188+0000 36'8 59:78 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865174+0000 36'7 59:78 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:49:03.256342+0000 2 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 0 0 0 1311 0 0 6 0 6 active+clean 2024-04-25T07:21:43.867977+0000 36'6 59:76 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 periodic scrub scheduled @ 2024-04-25T07:21:48.902280+0000 0 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.873469+0000 36'8 59:74 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:28:36.113620+0000 1 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 0 0 0 3702 0 0 10 0 10 active+clean 2024-04-25T07:21:43.869961+0000 36'10 59:93 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:44:01.652488+0000 2 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:43.876190+0000 36'6 59:72 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:40:22.443601+0000 1 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 0 0 0 3085 0 0 9 0 9 active+clean 2024-04-25T07:21:43.863330+0000 36'9 59:80 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:09:58.082452+0000 2 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:21:44.332657+0000 36'5 59:83 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:00:56.323139+0000 0 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.867331+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:17:56.553200+0000 2 0 2024-04-25T07:21:50.400 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:43.872600+0000 36'6 59:78 [2,0,1] 2 [2,0,1] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:21:27.343352+0000 0 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:44.425195+0000 36'6 59:83 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:34:36.618045+0000 2 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.862708+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:25:49.991290+0000 0 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.866899+0000 36'7 59:89 [2,1,0] 2 [2,1,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:37:19.724657+0000 1 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 1 1 0 0 2818 0 0 8 0 8 active+recovering+degraded 2024-04-25T07:21:43.878423+0000 52'8 59:78 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.865897+0000 36'8 59:91 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:11:09.161814+0000 2 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:44.334096+0000 36'7 59:85 [0,2,1] 0 [0,2,1] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T11:49:31.446794+0000 0 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 1 0 0 1851 0 0 6 0 6 active+recovery_wait+degraded 2024-04-25T07:21:44.426602+0000 36'6 59:83 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 1 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:21:44.425385+0000 36'5 59:82 [0,1,2] 0 [0,1,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:55:20.539024+0000 1 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.876826+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:12:11.691007+0000 1 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.864778+0000 36'7 59:89 [2,0,1] 2 [2,0,1] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T17:43:56.123745+0000 1 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.865808+0000 36'7 59:73 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:57:08.864139+0000 2 0 2024-04-25T07:21:50.401 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.864381+0000 36'8 59:74 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:55:02.937326+0000 2 0 2024-04-25T07:21:50.402 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:21:44.424933+0000 34'5 59:92 [0,1,2] 0 [0,1,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:25:29.983489+0000 1 0 2024-04-25T07:21:50.402 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:43.874772+0000 36'7 59:79 [2,1,0] 2 [2,1,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:25:33.019843+0000 1 0 2024-04-25T07:21:50.402 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:43.866635+0000 36'6 59:72 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T14:31:01.486204+0000 1 0 2024-04-25T07:21:50.402 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:43.873709+0000 36'8 59:79 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:48:07.118691+0000 3 0 2024-04-25T07:21:50.402 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 1 0 0 3085 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:44.427147+0000 36'8 59:85 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:21:50.402 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:21:50.402 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:21:50.402 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: activate_osd td/divergent-priors 1 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:21:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:21:50.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:21:50.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:21:50.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:21:50.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:21:50.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:21:50.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:21:50.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:21:50.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:21:50.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:21:50.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:21:50.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:21:50.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:21:50.419 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:21:50.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:21:50.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:21:50.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:21:50.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:21:50.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:21:50.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:21:50.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:21:50.463 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:50.469+0000 7f0c1a7d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:50.466 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:50.473+0000 7f0c1a7d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:50.467 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:50.473+0000 7f0c1a7d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:21:50.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:21:50.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:50.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:21:50.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:50.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:50.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:50.852 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:21:50.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:50.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:50.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:21:51.025 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:51.033+0000 7f0c1a7d37c0 -1 Falling back to public interface 2024-04-25T07:21:51.268 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 38 last_clean_interval [10,36) [v2:127.0.0.1:6818/1546537953,v1:127.0.0.1:6819/1546537953] [v2:127.0.0.1:6820/1546537953,v1:127.0.0.1:6821/1546537953] exists,up 8ebabd29-af92-4d13-bc6b-3f0ab12e5b80 2024-04-25T07:21:51.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:51.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:51.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:51.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: wait_for_osd up 1 2024-04-25T07:21:51.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:21:51.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:21:51.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:21:51.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:21:51.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:21:51.270 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:21:51.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:21:51.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:21:51.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:21:51.736 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 38 last_clean_interval [10,36) [v2:127.0.0.1:6818/1546537953,v1:127.0.0.1:6819/1546537953] [v2:127.0.0.1:6820/1546537953,v1:127.0.0.1:6821/1546537953] exists,up 8ebabd29-af92-4d13-bc6b-3f0ab12e5b80 2024-04-25T07:21:51.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:21:51.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:21:51.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:21:51.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: sleep 20 2024-04-25T07:21:53.961 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:21:53.969+0000 7f0c1a7d37c0 -1 osd.1 59 log_to_monitors true 2024-04-25T07:22:11.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: get_asok_path osd.1 2024-04-25T07:22:11.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:22:11.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:22:11.741 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:22:11.741 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:11.742 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:11.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.1.asok 2024-04-25T07:22:11.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: CEPH_ARGS= 2024-04-25T07:22:11.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:807: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.1.asok dump_ops_in_flight 2024-04-25T07:22:11.858 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:22:11.858 INFO:tasks.workunit.client.0.smithi134.stdout: "ops": [], 2024-04-25T07:22:11.858 INFO:tasks.workunit.client.0.smithi134.stdout: "num_ops": 0 2024-04-25T07:22:11.858 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:22:11.871 INFO:tasks.workunit.client.0.smithi134.stdout:allowing recovery 2024-04-25T07:22:11.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:809: TEST_divergent_3: echo 'allowing recovery' 2024-04-25T07:22:11.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:22:12.282 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:22:12.282 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.150852+0000 36'7 64:110 [0,2,1] 0 [0,2,1] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:30:29.781858+0000 1 0 2024-04-25T07:22:12.282 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.160412+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:56:59.888471+0000 1 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:56.163155+0000 36'8 64:84 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:34:49.078639+0000 0 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.167744+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:33:40.755480+0000 1 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 0 0 0 1928 0 0 7 0 7 active+clean 2024-04-25T07:21:56.162891+0000 36'7 64:87 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 1 1 0 0 3085 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:56.164955+0000 36'8 64:87 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.159007+0000 36'7 64:88 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:46:18.344940+0000 2 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 0 0 0 1311 0 0 6 0 6 active+clean 2024-04-25T07:21:56.161749+0000 36'6 64:86 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 5 0 0 0 0 3085 0 0 9 0 9 active+clean 2024-04-25T07:21:56.165880+0000 59'9 64:86 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:44:47.236022+0000 1 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 0 0 0 3702 0 0 10 0 10 active+clean 2024-04-25T07:21:56.159205+0000 36'10 64:106 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:01:05.771679+0000 2 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:56.163873+0000 36'6 64:82 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:46:09.998481+0000 1 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 0 0 0 3085 0 0 9 0 9 active+clean 2024-04-25T07:21:56.155864+0000 36'9 64:90 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:39:29.594454+0000 2 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:21:56.151561+0000 36'5 64:97 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:43:55.994173+0000 0 0 2024-04-25T07:22:12.283 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.161048+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:05:50.177552+0000 2 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:56.149348+0000 36'6 64:91 [2,0,1] 2 [2,0,1] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:53:30.492821+0000 0 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:56.152994+0000 36'6 64:97 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:03:22.377377+0000 2 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.159385+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:39:42.745638+0000 0 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.149240+0000 36'7 64:102 [2,1,0] 2 [2,1,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:30:50.267160+0000 1 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 1 1 0 0 2818 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:56.162099+0000 52'8 64:86 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:56.158189+0000 36'8 64:104 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:37:31.017474+0000 2 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.151460+0000 36'7 64:99 [0,2,1] 0 [0,2,1] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:06:33.697828+0000 0 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 1 0 0 1851 0 0 6 0 6 active+recovering+degraded 2024-04-25T07:21:56.168199+0000 36'6 64:97 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 1 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:21:56.153619+0000 36'5 64:96 [0,1,2] 0 [0,1,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:31:29.236961+0000 1 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.164550+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:06:03.365954+0000 1 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.018597+0000 36'7 64:102 [2,0,1] 2 [2,0,1] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:26:09.352827+0000 1 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.165204+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:31:35.583020+0000 2 0 2024-04-25T07:22:12.284 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:56.160370+0000 36'8 64:84 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:13:45.856082+0000 2 0 2024-04-25T07:22:12.285 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:21:56.152354+0000 34'5 64:106 [0,1,2] 0 [0,1,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:19:56.450338+0000 1 0 2024-04-25T07:22:12.285 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.148961+0000 36'7 64:92 [2,1,0] 2 [2,1,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:22:52.021535+0000 1 0 2024-04-25T07:22:12.285 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:56.164326+0000 36'6 64:82 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:19:49.507058+0000 1 0 2024-04-25T07:22:12.285 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:56.166597+0000 36'8 64:89 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:41:26.382250+0000 3 0 2024-04-25T07:22:12.285 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 1 0 0 3085 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:56.152386+0000 36'8 64:98 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:22:12.285 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:22:12.285 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:22:12.285 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:22:12.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:22:12.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:22:12.447 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:22:12.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:22:12.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:22:12.614 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:22:12.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: for i in $osds 2024-04-25T07:22:12.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:22:12.777 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:22:12.791 INFO:tasks.workunit.client.0.smithi134.stdout:reading divergent objects 2024-04-25T07:22:12.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:817: TEST_divergent_3: echo 'reading divergent objects' 2024-04-25T07:22:12.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:818: TEST_divergent_3: ceph pg dump pgs 2024-04-25T07:22:13.177 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:22:13.177 INFO:tasks.workunit.client.0.smithi134.stdout:1.1f 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.150852+0000 36'7 64:110 [0,2,1] 0 [0,2,1] 0 34'5 2024-04-25T07:19:35.258189+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:30:29.781858+0000 1 0 2024-04-25T07:22:13.177 INFO:tasks.workunit.client.0.smithi134.stdout:1.1e 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.160412+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:32.046633+0000 34'5 2024-04-25T07:19:32.046633+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:56:59.888471+0000 1 0 2024-04-25T07:22:13.177 INFO:tasks.workunit.client.0.smithi134.stdout:1.1d 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:56.163155+0000 36'8 64:84 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:35.031905+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:34:49.078639+0000 0 0 2024-04-25T07:22:13.177 INFO:tasks.workunit.client.0.smithi134.stdout:1.b 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.167744+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:33.104513+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:33:40.755480+0000 1 0 2024-04-25T07:22:13.177 INFO:tasks.workunit.client.0.smithi134.stdout:1.a 3 0 0 0 0 1928 0 0 7 0 7 active+clean 2024-04-25T07:21:56.162891+0000 36'7 64:87 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:43.149505+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:22:13.177 INFO:tasks.workunit.client.0.smithi134.stdout:1.9 5 1 1 0 0 3085 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:56.164955+0000 36'8 64:87 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:44.150350+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:22:13.177 INFO:tasks.workunit.client.0.smithi134.stdout:1.8 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.159007+0000 36'7 64:88 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:46.140639+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:46:18.344940+0000 2 0 2024-04-25T07:22:13.177 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 2 0 0 0 0 1311 0 0 6 0 6 active+clean 2024-04-25T07:21:56.161749+0000 36'6 64:86 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:42.218950+0000 0'0 2024-04-25T07:19:11.050015+0000 0 0 queued for scrub 0 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 5 0 0 0 0 3085 0 0 9 0 9 active+clean 2024-04-25T07:21:56.165880+0000 59'9 64:86 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:38.235328+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:44:47.236022+0000 1 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 6 0 0 0 0 3702 0 0 10 0 10 active+clean 2024-04-25T07:21:56.159205+0000 36'10 64:106 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:35.509739+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:01:05.771679+0000 2 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:56.163873+0000 36'6 64:82 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:29.130260+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T09:46:09.998481+0000 1 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 5 0 0 0 0 3085 0 0 9 0 9 active+clean 2024-04-25T07:21:56.155864+0000 36'9 64:90 [1,2,0] 1 [1,2,0] 1 34'6 2024-04-25T07:19:30.110163+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T13:39:29.594454+0000 2 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:21:56.151561+0000 36'5 64:97 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:33.675543+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:43:55.994173+0000 0 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.c 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.161048+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:40.079477+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:05:50.177552+0000 2 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:56.149348+0000 36'6 64:91 [2,0,1] 2 [2,0,1] 2 32'4 2024-04-25T07:19:34.501288+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:53:30.492821+0000 0 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.1c 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:56.152994+0000 36'6 64:97 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:40.341213+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:03:22.377377+0000 2 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.159385+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 32'4 2024-04-25T07:19:27.177221+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:39:42.745638+0000 0 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.d 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.149240+0000 36'7 64:102 [2,1,0] 2 [2,1,0] 2 34'5 2024-04-25T07:19:38.459919+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T12:30:50.267160+0000 1 0 2024-04-25T07:22:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:1.e 4 1 1 0 0 2818 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:56.162099+0000 52'8 64:86 [1,2,0] 1 [1,2,0] 1 36'6 2024-04-25T07:19:45.317352+0000 36'6 2024-04-25T07:19:45.317352+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.f 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:56.158189+0000 36'8 64:104 [2,0,1] 2 [2,0,1] 2 34'6 2024-04-25T07:19:33.676051+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:37:31.017474+0000 2 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.10 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.151460+0000 36'7 64:99 [0,2,1] 0 [0,2,1] 0 32'4 2024-04-25T07:19:34.281312+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:06:33.697828+0000 0 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.11 3 0 1 0 0 1851 0 0 6 0 6 active+recovering+degraded 2024-04-25T07:21:56.168199+0000 36'6 64:97 [0,1,2] 0 [0,1,2] 0 32'4 2024-04-25T07:19:36.287628+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 1 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.12 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:21:56.153619+0000 36'5 64:96 [0,1,2] 0 [0,1,2] 0 36'5 2024-04-25T07:19:39.267612+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:31:29.236961+0000 1 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.13 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.164550+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 36'5 2024-04-25T07:19:37.103298+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:06:03.365954+0000 1 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.14 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.018597+0000 36'7 64:102 [2,0,1] 2 [2,0,1] 2 34'5 2024-04-25T07:19:36.557105+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:26:09.352827+0000 1 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.15 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.165204+0000 36'7 64:83 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:36.173308+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T16:31:35.583020+0000 2 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.16 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:56.160370+0000 36'8 64:84 [1,0,2] 1 [1,0,2] 1 36'6 2024-04-25T07:19:39.172203+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T15:13:45.856082+0000 2 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.17 1 0 0 0 0 617 0 0 5 0 5 active+clean 2024-04-25T07:21:56.152354+0000 34'5 64:106 [0,1,2] 0 [0,1,2] 0 34'5 2024-04-25T07:19:38.218406+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T10:19:56.450338+0000 1 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.18 3 0 0 0 0 1851 0 0 7 0 7 active+clean 2024-04-25T07:21:56.148961+0000 36'7 64:92 [2,1,0] 2 [2,1,0] 2 36'5 2024-04-25T07:19:37.455368+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T18:22:52.021535+0000 1 0 2024-04-25T07:22:13.179 INFO:tasks.workunit.client.0.smithi134.stdout:1.19 2 0 0 0 0 1234 0 0 6 0 6 active+clean 2024-04-25T07:21:56.164326+0000 36'6 64:82 [1,0,2] 1 [1,0,2] 1 34'5 2024-04-25T07:19:34.113269+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T08:19:49.507058+0000 1 0 2024-04-25T07:22:13.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.1a 4 0 0 0 0 2468 0 0 8 0 8 active+clean 2024-04-25T07:21:56.166597+0000 36'8 64:89 [1,2,0] 1 [1,2,0] 1 36'7 2024-04-25T07:19:41.172230+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 periodic scrub scheduled @ 2024-04-26T07:41:26.382250+0000 3 0 2024-04-25T07:22:13.180 INFO:tasks.workunit.client.0.smithi134.stdout:1.1b 5 0 1 0 0 3085 0 0 8 0 8 active+recovery_wait+degraded 2024-04-25T07:21:56.152386+0000 36'8 64:98 [0,1,2] 0 [0,1,2] 0 36'6 2024-04-25T07:19:37.321615+0000 0'0 2024-04-25T07:19:11.050015+0000 0 1 no scrub is scheduled 3 0 2024-04-25T07:22:13.180 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:22:13.180 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:22:13.180 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:22:13.192 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: expr 2 + 2 2024-04-25T07:22:13.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: seq 1 4 2024-04-25T07:22:13.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:22:13.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:22:13.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:22:13.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:22:13.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:22:13.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:22:13.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:819: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:22:13.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:821: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:22:13.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: seq 1 2 2024-04-25T07:22:13.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:22:13.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2024-04-25T07:22:13.404 INFO:tasks.workunit.client.0.smithi134.stderr:error getting test/newobject_1: (2) No such file or directory 2024-04-25T07:22:13.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:823: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2024-04-25T07:22:13.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:825: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2024-04-25T07:22:13.444 INFO:tasks.workunit.client.0.smithi134.stderr:error getting test/newobject_2: (2) No such file or directory 2024-04-25T07:22:13.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: rm -f td/divergent-priors/existing 2024-04-25T07:22:13.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:22:13.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:22:13.462 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 36'9 (32'4) delete 1:70b3bf12:::existing_4:head by client.4633.0:1 2024-04-25T07:19:53.155268+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:22:13.462 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 20 _merge_object_divergent_entries: keeping 36'9 (32'4) delete 1:70b3bf12:::existing_4:head by client.4633.0:1 2024-04-25T07:19:53.155268+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 32'4 first_divergent_update: 36'9 last_divergent_update: 36'9 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 10 _merge_object_divergent_entries: more recent entry found: 52'8 (32'4) modify 1:70b3bf12:::existing_4:head by client.4754.0:1 2024-04-25T07:20:56.930634+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false, already merged 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 36'8 (0'0) modify 1:75ac5091:::newobject_2:head by client.4630.0:1 2024-04-25T07:19:53.146142+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 20 _merge_object_divergent_entries: keeping 36'8 (0'0) modify 1:75ac5091:::newobject_2:head by client.4630.0:1 2024-04-25T07:19:53.146142+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 36'8 last_divergent_update: 36'8 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.617+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.625+0000 7f2e606f1640 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 36'9 (32'2) modify 1:97db2a0f:::existing_2:head by client.4628.0:1 2024-04-25T07:19:53.140938+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.625+0000 7f2e606f1640 20 _merge_object_divergent_entries: keeping 36'9 (32'2) modify 1:97db2a0f:::existing_2:head by client.4628.0:1 2024-04-25T07:19:53.140938+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:22:13.463 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.625+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:22:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.625+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 32'2 first_divergent_update: 36'9 last_divergent_update: 36'9 2024-04-25T07:22:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.625+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2024-04-25T07:22:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.625+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:22:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.625+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 36'9 2024-04-25T07:22:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.625+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 36'9 (32'2) modify 1:97db2a0f:::existing_2:head by client.4628.0:1 2024-04-25T07:19:53.140938+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:22:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:43.625+0000 7f2e606f1640 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2024-04-25T07:22:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 36'7 (32'3) delete 1:8881e6d8:::existing_3:head by client.4638.0:1 2024-04-25T07:19:53.160919+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:22:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 20 _merge_object_divergent_entries: keeping 36'7 (32'3) delete 1:8881e6d8:::existing_3:head by client.4638.0:1 2024-04-25T07:19:53.160919+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 32'3 first_divergent_update: 36'7 last_divergent_update: 36'7 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 36'7 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 36'7 (32'3) delete 1:8881e6d8:::existing_3:head by client.4638.0:1 2024-04-25T07:19:53.160919+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 36'10 (32'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-04-25T07:19:53.161791+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 20 _merge_object_divergent_entries: keeping 36'10 (32'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-04-25T07:19:53.161791+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:22:13.465 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 32'1 first_divergent_update: 36'10 last_divergent_update: 36'10 2024-04-25T07:22:13.471 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-04-25T07:22:13.471 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:22:13.471 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 36'10 2024-04-25T07:22:13.471 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 36'10 (32'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-04-25T07:19:53.161791+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:22:13.471 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 36'9 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-04-25T07:19:53.134550+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 20 _merge_object_divergent_entries: keeping 36'9 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-04-25T07:19:53.134550+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 36'9 last_divergent_update: 36'9 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:21:44.229+0000 7f2e616f3640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.617+0000 7fc5d9a0a640 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 36'7 (32'3) delete 1:8881e6d8:::existing_3:head by client.4638.0:1 2024-04-25T07:19:53.160919+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.617+0000 7fc5d9a0a640 20 _merge_object_divergent_entries: keeping 36'7 (32'3) delete 1:8881e6d8:::existing_3:head by client.4638.0:1 2024-04-25T07:19:53.160919+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.617+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2024-04-25T07:22:13.472 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.617+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 32'3 first_divergent_update: 36'7 last_divergent_update: 36'7 2024-04-25T07:22:13.473 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.617+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2024-04-25T07:22:13.473 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.617+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:22:13.473 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.617+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 36'7 2024-04-25T07:22:13.473 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.617+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 36'7 (32'3) delete 1:8881e6d8:::existing_3:head by client.4638.0:1 2024-04-25T07:19:53.160919+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: false, new_object: false 2024-04-25T07:22:13.473 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.617+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2024-04-25T07:22:13.473 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 36'10 (32'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-04-25T07:19:53.161791+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:22:13.473 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 20 _merge_object_divergent_entries: keeping 36'10 (32'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-04-25T07:19:53.161791+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:22:13.473 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:22:13.473 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 32'1 first_divergent_update: 36'10 last_divergent_update: 36'10 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 36'10 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 36'10 (32'1) modify 1:d858c035:::existing_1:head by client.4639.0:1 2024-04-25T07:19:53.161791+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 36'9 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-04-25T07:19:53.134550+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 20 _merge_object_divergent_entries: keeping 36'9 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4624.0:1 2024-04-25T07:19:53.134550+0000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 36'9 last_divergent_update: 36'9 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2024-04-25T07:22:13.474 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.0.log:2024-04-25T07:21:43.621+0000 7fc5d9a0a640 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2024-04-25T07:22:13.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:22:13.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:22:13.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2024-04-25T07:22:13.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:837: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:22:13.503 INFO:tasks.workunit.client.0.smithi134.stdout:success 2024-04-25T07:22:13.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:842: TEST_divergent_3: echo success 2024-04-25T07:22:13.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:844: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.26105.out 2024-04-25T07:22:13.504 INFO:tasks.workunit.client.0.smithi134.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.26105.out': No such file or directory 2024-04-25T07:22:13.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:846: TEST_divergent_3: delete_pool test 2024-04-25T07:22:13.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:22:13.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:22:14.121 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T07:22:14.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:847: TEST_divergent_3: kill_daemons td/divergent-priors 2024-04-25T07:22:14.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:22:14.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:22:14.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:22:14.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:22:14.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:22:19.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:22:19.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:22:19.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:22:19.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:22:19.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:22:19.460 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:22:19.460 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:22:19.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:22:19.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:22:19.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:22:19.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:22:19.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:22:19.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:22:19.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:22:19.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:22:19.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:22:19.473 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:22:19.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:22:19.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:22:19.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:22:19.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:22:19.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:22:19.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:22:19.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:22:19.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:22:19.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:19.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:19.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26105 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:22:19.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:22:19.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:22:19.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:22:19.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:22:19.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:22:19.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:22:19.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:22:19.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:22:19.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:22:19.523 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:22:19.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:22:19.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:22:19.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:22:19.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:22:19.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:22:19.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:22:19.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:22:19.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:22:19.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:19.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26105 2024-04-25T07:22:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:22:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:22:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2024-04-25T07:22:19.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:22:19.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:19.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:19.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.26105 2024-04-25T07:22:19.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:22:19.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:22:19.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:22:19.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2024-04-25T07:22:19.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2024-04-25T07:22:19.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2024-04-25T07:22:19.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2024-04-25T07:22:19.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2024-04-25T07:22:19.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2024-04-25T07:22:19.535 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2024-04-25T07:22:19.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2024-04-25T07:22:19.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2024-04-25T07:22:19.535 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:22:19.535 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:22:19.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2024-04-25T07:22:19.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2024-04-25T07:22:19.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:22:19.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:22:19.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:22:19.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2024-04-25T07:22:19.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2024-04-25T07:22:19.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:22:19.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:19.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:19.986 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:19.986 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:19.986 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:19.987 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:22:19.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:22:20.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:22:20.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:22:20.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:22:20.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:22:20.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:22:20.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:22:20.225 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:22:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:22:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:22:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:22:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-mon.a.asok 2024-04-25T07:22:20.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:22:20.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26105/ceph-mon.a.asok config get fsid 2024-04-25T07:22:20.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:22:20.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:22:20.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:22:20.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:22:20.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:22:20.345 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:22:20.345 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:22:20.345 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:22:20.345 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:22:20.345 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:20.345 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:20.346 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-mon.a.asok 2024-04-25T07:22:20.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:22:20.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.26105/ceph-mon.a.asok config get mon_host 2024-04-25T07:22:20.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2024-04-25T07:22:20.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/divergent-priors 2024-04-25T07:22:20.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:22:20.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:22:20.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:22:20.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/divergent-priors/x 2024-04-25T07:22:20.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:22:21.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:22:21.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:21.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:21.371 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:21.371 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:21.371 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:21.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:22:21.372 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:22:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:22:21.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:22:21.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2024-04-25T07:22:21.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:22:21.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:22:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:22:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:22:21.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:21.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:21.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:21.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:21.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:22:21.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:22:21.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:22:21.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:22:21.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c09e27c0-df59-4ace-bcfa-29969022462f 2024-04-25T07:22:21.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c09e27c0-df59-4ace-bcfa-29969022462f' 2024-04-25T07:22:21.418 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 c09e27c0-df59-4ace-bcfa-29969022462f 2024-04-25T07:22:21.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:22:21.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCtBCpmTXqpGhAA8v2M3D7jSzU5TrDIWGMgbw== 2024-04-25T07:22:21.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCtBCpmTXqpGhAA8v2M3D7jSzU5TrDIWGMgbw=="}' 2024-04-25T07:22:21.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c09e27c0-df59-4ace-bcfa-29969022462f -i td/divergent-priors/0/new.json 2024-04-25T07:22:21.744 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:22:21.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/0/new.json 2024-04-25T07:22:21.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCtBCpmTXqpGhAA8v2M3D7jSzU5TrDIWGMgbw== --osd-uuid c09e27c0-df59-4ace-bcfa-29969022462f 2024-04-25T07:22:21.804 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:21.812+0000 7f14811107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:21.811 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:21.816+0000 7f14811107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:21.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:21.820+0000 7f14811107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:21.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:21.820+0000 7f14811107c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:22:21.814 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:21.820+0000 7f14811107c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:22:21.814 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:21.820+0000 7f14811107c0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2024-04-25T07:22:21.852 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:21.856+0000 7f14811107c0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2024-04-25T07:22:28.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/0/keyring 2024-04-25T07:22:28.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:22:28.192 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:22:28.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:22:28.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:22:28.759 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:22:28.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:22:28.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:22:28.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:22:28.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:22:28.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:22:28.799 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:28.804+0000 7fdc71b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:28.843 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:28.848+0000 7fdc71b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:28.844 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:28.852+0000 7fdc71b9d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:29.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:22:29.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:22:29.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:22:29.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:22:29.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:22:29.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:29.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:22:29.234 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:22:29.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:29.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:29.419 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:29.424+0000 7fdc71b9d7c0 -1 Falling back to public interface 2024-04-25T07:22:29.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:30.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:30.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:30.711 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:22:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:22:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:31.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:32.058 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:32.063+0000 7fdc71b9d7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:22:32.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:32.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:32.137 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:22:32.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:22:32.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:32.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:32.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:33.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:33.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:33.803 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:22:33.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:22:33.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:33.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:34.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:35.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:35.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:35.243 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:22:35.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:22:35.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:35.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:35.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:36.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:36.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:36.682 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:22:36.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:22:36.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:36.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:37.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:38.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:38.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:38.227 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:22:38.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:22:38.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:38.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:38.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:39.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:39.686 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:22:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:22:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:39.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:40.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:41.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:41.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:41.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:22:41.259 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:22:41.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:41.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:41.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:42.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:42.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:42.737 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:22:42.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:22:42.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:42.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:43.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:44.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:44.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:44.258 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:22:44.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:22:44.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:44.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:44.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:45.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:45.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:45.727 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:22:45.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:22:45.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:45.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:46.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:47.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:47.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:47.824 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:22:47.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:22:47.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:47.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:49.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:50.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:50.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:50.169 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:22:50.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:22:50.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:50.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:51.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:52.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:52.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:52.167 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:22:52.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:22:52.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:52.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:53.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:54.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:54.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:54.318 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:22:54.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:22:54.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:54.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:55.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:56.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:56.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:56.069 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:22:56.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:22:56.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:56.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:57.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:22:58.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:22:58.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:22:58.072 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:22:58.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:22:58.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:22:58.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:22:58.510 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3050493816,v1:127.0.0.1:6803/3050493816] [v2:127.0.0.1:6804/3050493816,v1:127.0.0.1:6805/3050493816] exists,up c09e27c0-df59-4ace-bcfa-29969022462f 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:22:58.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:22:58.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:22:58.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:22:58.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=87b81a46-8982-455b-8e0b-0fac08f51bc8 2024-04-25T07:22:58.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 87b81a46-8982-455b-8e0b-0fac08f51bc8' 2024-04-25T07:22:58.518 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 87b81a46-8982-455b-8e0b-0fac08f51bc8 2024-04-25T07:22:58.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:22:58.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSBCpmiPWTIBAADrQquvsc/Ak/3cFnkfitkg== 2024-04-25T07:22:58.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSBCpmiPWTIBAADrQquvsc/Ak/3cFnkfitkg=="}' 2024-04-25T07:22:58.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 87b81a46-8982-455b-8e0b-0fac08f51bc8 -i td/divergent-priors/1/new.json 2024-04-25T07:22:59.048 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:22:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/1/new.json 2024-04-25T07:22:59.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSBCpmiPWTIBAADrQquvsc/Ak/3cFnkfitkg== --osd-uuid 87b81a46-8982-455b-8e0b-0fac08f51bc8 2024-04-25T07:22:59.105 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:59.111+0000 7f7d603cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:59.116 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:59.123+0000 7f7d603cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:59.118 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:59.123+0000 7f7d603cf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:22:59.118 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:59.123+0000 7f7d603cf7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:22:59.118 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:59.123+0000 7f7d603cf7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:22:59.118 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:59.123+0000 7f7d603cf7c0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2024-04-25T07:22:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:22:59.151+0000 7f7d603cf7c0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2024-04-25T07:23:05.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/1/keyring 2024-04-25T07:23:05.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:23:05.605 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:23:05.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:23:05.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:23:06.216 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:23:06.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:23:06.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:23:06.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:23:06.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:23:06.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:23:06.256 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:06.258+0000 7f05cf3a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:06.279 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:06.282+0000 7f05cf3a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:06.280 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:06.286+0000 7f05cf3a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:06.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:23:06.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:23:06.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:23:06.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:23:06.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:23:06.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:06.674 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:23:06.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:23:06.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:06.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:06.862 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:06.866+0000 7f05cf3a37c0 -1 Falling back to public interface 2024-04-25T07:23:07.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:08.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:08.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:08.099 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:23:08.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:23:08.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:08.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:08.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:09.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:09.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:09.522 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:23:09.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:23:09.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:09.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:09.562 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:09.566+0000 7f05cf3a37c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:23:10.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:11.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:11.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:11.120 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:23:11.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:23:11.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:11.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:11.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:12.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:12.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:12.595 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:23:12.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:23:12.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:12.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:13.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:14.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:14.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:14.161 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:23:14.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:23:14.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:14.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:14.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:15.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:15.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:15.807 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:23:15.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:23:15.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:15.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:16.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:17.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:17.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:17.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:23:17.273 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:23:17.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:17.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:17.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:18.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:18.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:18.794 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:23:18.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:23:18.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:18.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:19.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:20.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:20.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:20.243 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:23:20.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:23:20.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:20.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:20.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:21.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:21.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:21.867 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:23:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:23:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:22.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:23.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:23.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:23.474 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:23:23.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:23:23.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:23.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:24.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:25.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:25.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:25.402 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:23:25.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:23:25.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:25.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:26.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:27.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:27.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:27.241 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:23:27.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:23:27.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:27.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:28.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:29.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:29.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:29.080 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:23:29.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:23:29.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:29.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:29.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:30.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:30.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:30.932 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:23:30.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:23:30.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:30.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:32.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:33.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:33.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:33.100 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:23:33.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:23:33.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:33.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:33.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:34.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:34.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:34.910 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:23:34.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:23:34.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:34.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:23:35.415 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3821439290,v1:127.0.0.1:6811/3821439290] [v2:127.0.0.1:6812/3821439290,v1:127.0.0.1:6813/3821439290] exists,up 87b81a46-8982-455b-8e0b-0fac08f51bc8 2024-04-25T07:23:35.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:23:35.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:23:35.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:23:35.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2024-04-25T07:23:35.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2024-04-25T07:23:35.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/divergent-priors 2024-04-25T07:23:35.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:23:35.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:23:35.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:23:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:23:35.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:23:35.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:23:35.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:23:35.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a22a9735-1827-49a9-ad2a-a2ffcbe059d8 2024-04-25T07:23:35.422 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 a22a9735-1827-49a9-ad2a-a2ffcbe059d8 2024-04-25T07:23:35.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a22a9735-1827-49a9-ad2a-a2ffcbe059d8' 2024-04-25T07:23:35.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:23:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD3BCpmQLj6GhAAhjCVXpKQU14/zB3eNs9N/Q== 2024-04-25T07:23:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD3BCpmQLj6GhAAhjCVXpKQU14/zB3eNs9N/Q=="}' 2024-04-25T07:23:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a22a9735-1827-49a9-ad2a-a2ffcbe059d8 -i td/divergent-priors/2/new.json 2024-04-25T07:23:35.947 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:23:35.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/divergent-priors/2/new.json 2024-04-25T07:23:35.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD3BCpmQLj6GhAAhjCVXpKQU14/zB3eNs9N/Q== --osd-uuid a22a9735-1827-49a9-ad2a-a2ffcbe059d8 2024-04-25T07:23:36.008 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:36.013+0000 7f177b7507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:36.081 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:36.089+0000 7f177b7507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:36.083 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:36.089+0000 7f177b7507c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:36.083 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:36.089+0000 7f177b7507c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:23:36.083 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:36.089+0000 7f177b7507c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:23:36.083 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:36.089+0000 7f177b7507c0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2024-04-25T07:23:36.205 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:36.213+0000 7f177b7507c0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2024-04-25T07:23:42.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/divergent-priors/2/keyring 2024-04-25T07:23:42.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:23:42.554 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:23:42.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:23:42.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:23:43.123 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:23:43.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:23:43.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:23:43.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:23:43.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:23:43.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:23:43.163 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:43.169+0000 7fdbd66187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:43.206 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:43.213+0000 7fdbd66187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:43.208 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:43.213+0000 7fdbd66187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:23:43.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:23:43.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:23:43.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:23:43.612 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:23:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:23:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:23:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:23:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:43.798 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:43.805+0000 7fdbd66187c0 -1 Falling back to public interface 2024-04-25T07:23:44.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:45.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:45.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:45.026 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:23:45.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:23:45.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:45.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:45.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:46.450 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:23:46.453+0000 7fdbd66187c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:23:46.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:46.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:46.451 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:23:46.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:23:46.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:46.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:46.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:47.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:47.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:47.856 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:23:47.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:23:47.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:47.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:48.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:49.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:49.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:49.546 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:23:49.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:23:49.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:49.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:50.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:51.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:51.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:51.010 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:23:51.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:23:51.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:51.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:51.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:52.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:52.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:52.546 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:23:52.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:23:52.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:52.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:53.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:54.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:54.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:54.107 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:23:54.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:23:54.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:54.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:54.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:55.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:55.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:55.588 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:23:55.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:23:55.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:55.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:56.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:57.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:57.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:57.061 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:23:57.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:23:57.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:57.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:57.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:23:58.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:23:58.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:23:58.625 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:23:58.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:23:58.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:23:58.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:23:59.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:00.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:00.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:00.173 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:24:00.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:24:00.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:00.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:00.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:01.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:01.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:01.741 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:24:01.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:24:01.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:01.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:02.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:03.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:03.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:03.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:24:03.325 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:24:03.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:03.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:04.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:05.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:05.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:05.403 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:24:05.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:24:05.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:05.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:06.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:07.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:07.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:07.255 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:24:07.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:24:07.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:07.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:08.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:09.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:09.143 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:24:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:24:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:09.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:10.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:10.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:10.994 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:24:10.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:24:10.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:10.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:11.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:24:12.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:24:12.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:24:12.888 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:24:12.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:24:12.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:24:12.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:24:13.316 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1044975095,v1:127.0.0.1:6819/1044975095] [v2:127.0.0.1:6820/1044975095,v1:127.0.0.1:6821/1044975095] exists,up a22a9735-1827-49a9-ad2a-a2ffcbe059d8 2024-04-25T07:24:13.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:24:13.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:24:13.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:24:13.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2024-04-25T07:24:14.070 INFO:tasks.workunit.client.0.smithi134.stderr:noout is set 2024-04-25T07:24:14.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2024-04-25T07:24:14.748 INFO:tasks.workunit.client.0.smithi134.stderr:noin is set 2024-04-25T07:24:14.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2024-04-25T07:24:15.769 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T07:24:15.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2024-04-25T07:24:15.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2451: create_ec_pool: local pool_name=test 2024-04-25T07:24:15.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2452: create_ec_pool: shift 2024-04-25T07:24:15.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2453: create_ec_pool: local allow_overwrites=true 2024-04-25T07:24:15.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2454: create_ec_pool: shift 2024-04-25T07:24:15.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2456: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2024-04-25T07:24:16.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2458: create_ec_pool: create_pool test 1 1 erasure myprofile 2024-04-25T07:24:16.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T07:24:17.278 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T07:24:17.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:24:18.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2460: create_ec_pool: '[' true = true ']' 2024-04-25T07:24:18.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2461: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2024-04-25T07:24:19.543 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 allow_ec_overwrites to true 2024-04-25T07:24:19.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2464: create_ec_pool: wait_for_clean 2024-04-25T07:24:19.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:24:19.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:24:19.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:24:19.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:24:19.569 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:24:19.569 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:24:19.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:24:19.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:24:19.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:24:19.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:24:19.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:24:19.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:24:19.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:24:19.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:24:19.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:24:20.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:24:20.138 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:24:20.138 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:24:20.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:24:20.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:20.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:24:20.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836507 2024-04-25T07:24:20.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836507 2024-04-25T07:24:20.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507' 2024-04-25T07:24:20.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:20.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:24:20.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-04-25T07:24:20.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-04-25T07:24:20.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672979' 2024-04-25T07:24:20.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:20.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:24:20.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509452 2024-04-25T07:24:20.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509452 2024-04-25T07:24:20.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836507 1-42949672979 2-64424509452' 2024-04-25T07:24:20.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:20.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836507 2024-04-25T07:24:20.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:20.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:24:20.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836507 2024-04-25T07:24:20.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:20.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836507 2024-04-25T07:24:20.652 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836507 2024-04-25T07:24:20.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-04-25T07:24:20.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:24:21.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836505 -lt 21474836507 2024-04-25T07:24:21.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:24:22.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:24:22.071 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:24:22.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836507 2024-04-25T07:24:22.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:22.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-04-25T07:24:22.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:22.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:24:22.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-04-25T07:24:22.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:22.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-04-25T07:24:22.536 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672979 2024-04-25T07:24:22.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-04-25T07:24:22.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:24:22.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672979 2024-04-25T07:24:22.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:22.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509452 2024-04-25T07:24:22.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:22.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:24:22.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509452 2024-04-25T07:24:22.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:22.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509452 2024-04-25T07:24:22.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-04-25T07:24:22.973 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509452 2024-04-25T07:24:22.973 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:24:23.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509452 -lt 64424509452 2024-04-25T07:24:23.396 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:24:23.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:24:23.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:24:23.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:24:23.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:24:23.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:24:23.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:24:23.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:24:23.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:24:23.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:24:23.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:24:24.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:24:24.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:24:24.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:24:24.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:24:25.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:24:25.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:24:25.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:24:25.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2465: create_ec_pool: return 0 2024-04-25T07:24:25.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2024-04-25T07:24:25.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:24:25.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:24:25.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:24:25.463 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:24:25.463 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:24:25.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:24:25.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:25.463 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:24:25.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-04-25T07:24:25.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-04-25T07:24:25.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-04-25T07:24:25.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:25.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:24:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672982 2024-04-25T07:24:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672982 2024-04-25T07:24:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672982' 2024-04-25T07:24:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:25.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:24:25.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509455 2024-04-25T07:24:25.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509455 2024-04-25T07:24:25.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672982 2-64424509455' 2024-04-25T07:24:25.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:25.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:25.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-04-25T07:24:25.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:24:25.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-04-25T07:24:25.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:25.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-04-25T07:24:25.944 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836510 2024-04-25T07:24:25.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-04-25T07:24:25.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:24:26.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-04-25T07:24:26.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:26.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:26.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672982 2024-04-25T07:24:26.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:24:26.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672982 2024-04-25T07:24:26.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:26.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672982 2024-04-25T07:24:26.369 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672982 2024-04-25T07:24:26.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672982' 2024-04-25T07:24:26.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:24:26.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672982 2024-04-25T07:24:26.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:26.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509455 2024-04-25T07:24:26.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:26.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:24:26.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509455 2024-04-25T07:24:26.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:26.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509455 2024-04-25T07:24:26.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-04-25T07:24:26.814 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509455 2024-04-25T07:24:26.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:24:27.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509455 -lt 64424509455 2024-04-25T07:24:27.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2024-04-25T07:24:27.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:24:27.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:24:27.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:24:27.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:24:27.226 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:24:27.226 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:24:27.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:24:27.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:24:27.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:24:27.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:24:27.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:24:27.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:24:27.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:24:27.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:24:27.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:24:27.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:24:27.754 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:24:27.754 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:24:27.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:24:27.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:27.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:24:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836513 2024-04-25T07:24:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836513 2024-04-25T07:24:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513' 2024-04-25T07:24:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:27.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:24:28.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-04-25T07:24:28.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-04-25T07:24:28.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672985' 2024-04-25T07:24:28.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:28.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:24:28.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-04-25T07:24:28.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-04-25T07:24:28.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836513 1-42949672985 2-64424509458' 2024-04-25T07:24:28.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:28.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836513 2024-04-25T07:24:28.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:28.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:24:28.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836513 2024-04-25T07:24:28.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:28.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836513 2024-04-25T07:24:28.232 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836513 2024-04-25T07:24:28.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836513' 2024-04-25T07:24:28.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:24:28.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836513 -lt 21474836513 2024-04-25T07:24:28.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:28.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:28.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-04-25T07:24:28.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:24:28.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-04-25T07:24:28.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:28.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-04-25T07:24:28.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-25T07:24:28.653 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672985 2024-04-25T07:24:28.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:24:29.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672983 -lt 42949672985 2024-04-25T07:24:29.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:24:30.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:24:30.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:24:30.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-04-25T07:24:30.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:30.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:30.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-04-25T07:24:30.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:24:30.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-04-25T07:24:30.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:30.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-04-25T07:24:30.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-04-25T07:24:30.554 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509458 2024-04-25T07:24:30.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:24:30.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-04-25T07:24:30.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:24:30.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:24:30.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:24:31.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:24:31.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:24:31.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:24:31.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:24:31.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:24:31.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:24:31.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:24:31.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:24:31.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:24:32.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:24:32.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:24:32.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:24:32.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:24:32.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:24:32.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:24:32.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2024-04-25T07:24:32.565 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2024-04-25T07:24:32.965 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:24:32.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2024-04-25T07:24:32.981 INFO:tasks.workunit.client.0.smithi134.stdout:primary and soon to be divergent is 1 2024-04-25T07:24:32.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2024-04-25T07:24:32.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:24:33.356 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:24:33.356 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T07:24:19.040506+0000 0'0 29:15 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:52:18.138839+0000 0 0 2024-04-25T07:24:33.356 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:24:33.356 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:24:33.356 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2024-04-25T07:24:33.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2024-04-25T07:24:33.372 INFO:tasks.workunit.client.0.smithi134.stdout:writing initial objects 2024-04-25T07:24:33.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2024-04-25T07:24:33.372 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2024-04-25T07:24:33.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:33.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2024-04-25T07:24:33.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:33.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2024-04-25T07:24:33.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:33.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2024-04-25T07:24:34.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:34.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2024-04-25T07:24:34.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:34.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2024-04-25T07:24:34.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:34.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2024-04-25T07:24:34.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:34.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2024-04-25T07:24:35.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:35.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2024-04-25T07:24:35.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:35.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2024-04-25T07:24:35.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:35.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2024-04-25T07:24:35.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:35.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2024-04-25T07:24:35.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:35.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2024-04-25T07:24:36.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:36.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2024-04-25T07:24:36.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:36.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2024-04-25T07:24:36.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:36.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2024-04-25T07:24:36.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:36.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2024-04-25T07:24:37.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:37.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2024-04-25T07:24:37.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:37.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2024-04-25T07:24:37.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:37.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2024-04-25T07:24:38.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:38.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2024-04-25T07:24:38.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:38.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2024-04-25T07:24:38.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:38.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2024-04-25T07:24:38.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:38.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2024-04-25T07:24:39.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:39.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2024-04-25T07:24:39.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:39.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2024-04-25T07:24:39.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:39.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2024-04-25T07:24:39.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:39.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2024-04-25T07:24:39.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:39.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2024-04-25T07:24:40.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:40.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2024-04-25T07:24:40.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:40.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2024-04-25T07:24:40.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:40.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2024-04-25T07:24:40.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:40.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2024-04-25T07:24:41.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:41.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2024-04-25T07:24:41.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:41.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2024-04-25T07:24:41.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:41.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2024-04-25T07:24:41.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:41.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2024-04-25T07:24:42.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:42.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2024-04-25T07:24:42.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:42.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2024-04-25T07:24:42.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:42.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2024-04-25T07:24:42.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:42.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2024-04-25T07:24:43.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:43.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2024-04-25T07:24:43.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:43.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2024-04-25T07:24:43.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:43.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2024-04-25T07:24:43.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:43.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2024-04-25T07:24:44.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:44.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2024-04-25T07:24:44.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:44.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2024-04-25T07:24:44.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:44.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2024-04-25T07:24:44.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:44.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2024-04-25T07:24:45.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:45.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2024-04-25T07:24:45.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:45.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2024-04-25T07:24:45.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:45.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2024-04-25T07:24:45.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:45.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2024-04-25T07:24:46.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:46.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2024-04-25T07:24:46.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:46.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2024-04-25T07:24:46.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:46.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2024-04-25T07:24:46.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:46.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2024-04-25T07:24:47.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:47.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2024-04-25T07:24:47.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:47.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2024-04-25T07:24:47.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:47.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2024-04-25T07:24:47.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:47.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2024-04-25T07:24:48.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:48.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2024-04-25T07:24:48.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:48.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2024-04-25T07:24:48.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:48.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2024-04-25T07:24:48.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:48.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2024-04-25T07:24:49.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:49.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2024-04-25T07:24:49.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:49.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2024-04-25T07:24:49.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:49.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2024-04-25T07:24:49.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:49.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2024-04-25T07:24:50.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:50.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2024-04-25T07:24:50.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:50.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2024-04-25T07:24:50.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:50.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2024-04-25T07:24:50.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:50.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2024-04-25T07:24:51.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:51.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2024-04-25T07:24:51.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:51.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2024-04-25T07:24:51.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:51.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2024-04-25T07:24:51.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:51.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2024-04-25T07:24:52.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:52.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2024-04-25T07:24:52.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:52.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2024-04-25T07:24:52.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:52.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2024-04-25T07:24:52.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:52.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2024-04-25T07:24:53.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:53.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2024-04-25T07:24:53.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:53.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2024-04-25T07:24:53.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:53.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2024-04-25T07:24:53.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:53.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2024-04-25T07:24:53.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:53.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2024-04-25T07:24:54.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:54.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2024-04-25T07:24:54.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:54.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2024-04-25T07:24:54.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:54.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2024-04-25T07:24:55.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:55.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2024-04-25T07:24:55.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:55.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2024-04-25T07:24:55.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:55.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2024-04-25T07:24:55.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:55.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2024-04-25T07:24:56.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:56.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2024-04-25T07:24:56.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:56.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2024-04-25T07:24:56.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:56.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2024-04-25T07:24:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2024-04-25T07:24:56.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:56.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2024-04-25T07:24:57.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:57.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2024-04-25T07:24:57.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:57.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-04-25T07:24:57.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2024-04-25T07:24:57.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-04-25T07:24:57.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:24:57.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2024-04-25T07:24:57.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:24:57.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:24:57.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:24:57.998 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:24:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:24:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:24:57.998 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:24:57.998 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:24:57.998 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:24:58.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:24:58.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:24:58.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:24:58.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:24:58.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:24:58.078 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:24:58.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:24:58.505 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:24:58.505 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:24:58.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:24:58.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:58.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:24:58.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836521 2024-04-25T07:24:58.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836521 2024-04-25T07:24:58.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521' 2024-04-25T07:24:58.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:58.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:24:58.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672993 2024-04-25T07:24:58.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672993 2024-04-25T07:24:58.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521 1-42949672993' 2024-04-25T07:24:58.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:24:58.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:24:58.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509466 2024-04-25T07:24:58.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509466 2024-04-25T07:24:58.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836521 1-42949672993 2-64424509466' 2024-04-25T07:24:58.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:24:58.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836521 2024-04-25T07:24:58.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:24:58.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:24:58.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836521 2024-04-25T07:24:58.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:24:58.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836521 2024-04-25T07:24:58.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836521' 2024-04-25T07:24:58.984 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836521 2024-04-25T07:24:58.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:24:59.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836519 -lt 21474836521 2024-04-25T07:24:59.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:25:00.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:25:00.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:25:00.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836521 -lt 21474836521 2024-04-25T07:25:00.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:25:00.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672993 2024-04-25T07:25:00.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:25:00.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:25:00.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672993 2024-04-25T07:25:00.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:25:00.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672993 2024-04-25T07:25:00.831 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672993 2024-04-25T07:25:00.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672993' 2024-04-25T07:25:00.832 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:25:01.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672993 -lt 42949672993 2024-04-25T07:25:01.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:25:01.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509466 2024-04-25T07:25:01.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:25:01.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:25:01.271 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509466 2024-04-25T07:25:01.271 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:25:01.272 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509466 2024-04-25T07:25:01.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509466 2024-04-25T07:25:01.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509466' 2024-04-25T07:25:01.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:25:01.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509466 -lt 64424509466 2024-04-25T07:25:01.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:25:01.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:01.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:02.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:25:02.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:02.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:02.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:02.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:02.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:02.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:02.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:02.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:25:02.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:02.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:02.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:03.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:25:03.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:25:03.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:25:03.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2024-04-25T07:25:03.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T07:25:03.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=existing_1 2024-04-25T07:25:03.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test existing_1 2024-04-25T07:25:03.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T07:25:03.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2024-04-25T07:25:03.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2024-04-25T07:25:03.867 INFO:tasks.workunit.client.0.smithi134.stdout:blackholing osds 0 2 2024-04-25T07:25:03.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:25:04.263 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:25:04.264 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 0 0 0 61700 0 0 16 84 16 active+clean 2024-04-25T07:24:19.040506+0000 29'100 29:115 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:52:18.138839+0000 0 0 2024-04-25T07:25:04.264 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:25:04.264 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:25:04.264 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:25:04.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:04.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2024-04-25T07:25:04.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:25:04.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:25:04.280 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:25:04.280 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:04.280 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:25:04.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.0.asok 2024-04-25T07:25:04.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:25:04.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.0.asok config set objectstore_blackhole 1 2024-04-25T07:25:04.389 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:25:04.389 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:25:04.389 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:25:04.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:04.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2024-04-25T07:25:04.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:25:04.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:25:04.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:25:04.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:04.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:25:04.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.2.asok 2024-04-25T07:25:04.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:25:04.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.2.asok config set objectstore_blackhole 1 2024-04-25T07:25:04.507 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:25:04.507 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "objectstore_blackhole = '' (not observed, change may require restart) " 2024-04-25T07:25:04.507 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:25:04.519 INFO:tasks.workunit.client.0.smithi134.stdout:writing divergent object 2024-04-25T07:25:04.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2024-04-25T07:25:04.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2024-04-25T07:25:04.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2024-04-25T07:25:05.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2024-04-25T07:25:05.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2024-04-25T07:25:05.785 INFO:tasks.workunit.client.0.smithi134.stdout:created pool test snap snap1 2024-04-25T07:25:05.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2024-04-25T07:25:05.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2024-04-25T07:25:05.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2024-04-25T07:25:15.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2024-04-25T07:25:15.797 INFO:tasks.workunit.client.0.smithi134.stderr:rados: no process found 2024-04-25T07:25:15.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2024-04-25T07:25:15.797 INFO:tasks.workunit.client.0.smithi134.stdout:killing all the osds 2024-04-25T07:25:15.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:25:16.214 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:25:16.215 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 active+clean 2024-04-25T07:24:19.040506+0000 30'104 30:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:35:58.735459+0000 0 0 2024-04-25T07:25:16.215 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:25:16.215 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:25:16.215 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:25:16.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2024-04-25T07:25:16.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:25:16.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:25:16.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:25:16.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:25:16.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:25:16.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:25:16.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:25:16.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2024-04-25T07:25:17.123 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:25:17.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:25:17.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2024-04-25T07:25:17.733 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already down. 2024-04-25T07:25:17.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2024-04-25T07:25:17.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2024-04-25T07:25:18.309 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already down. 2024-04-25T07:25:18.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:18.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2024-04-25T07:25:18.941 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T07:25:18.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:18.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2024-04-25T07:25:19.554 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already out. 2024-04-25T07:25:19.570 INFO:tasks.workunit.client.0.smithi134.stdout:bringing up non_divergent 0 2 2024-04-25T07:25:19.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2024-04-25T07:25:19.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:25:20.034 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:25:20.034 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 0 0 0 62317 0 0 14 90 14 stale+active+clean 2024-04-25T07:24:19.040506+0000 30'104 30:121 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T07:35:58.735459+0000 0 0 2024-04-25T07:25:20.034 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:25:20.034 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:25:20.034 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:25:20.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:20.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-04-25T07:25:20.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:25:20.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:25:20.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:25:20.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:25:20.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:25:20.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:25:20.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:25:20.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:25:20.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:25:20.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:25:20.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:25:20.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:25:20.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:25:20.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:25:20.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:25:20.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:25:20.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:25:20.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:25:20.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:25:20.056 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:25:20.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:25:20.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:25:20.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:25:20.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:25:20.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:25:20.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:25:20.095 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:20.101+0000 7fdeb72967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:20.097 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:20.101+0000 7fdeb72967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:20.098 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:20.105+0000 7fdeb72967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:20.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:25:20.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:25:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:25:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:25:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:25:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:25:20.480 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:25:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:20.676 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:20.681+0000 7fdeb72967c0 -1 Falling back to public interface 2024-04-25T07:25:20.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:21.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:21.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:21.935 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:25:21.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:25:21.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:21.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:22.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:23.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:23.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:23.376 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:25:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:25:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:23.728 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:23.733+0000 7fdeb72967c0 -1 osd.0 29 log_to_monitors true 2024-04-25T07:25:23.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:24.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:24.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:24.796 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:25:24.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:25:24.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:24.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:25:25.221 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up out weight 0 up_from 37 up_thru 0 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/2626715406,v1:127.0.0.1:6803/2626715406] [v2:127.0.0.1:6804/2626715406,v1:127.0.0.1:6805/2626715406] exists,up c09e27c0-df59-4ace-bcfa-29969022462f 2024-04-25T07:25:25.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:25:25.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:25:25.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:25:25.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:25:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:25:25.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:25:25.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:25:25.228 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:25:25.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:25:25.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:25:25.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:25:25.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:25:25.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:25:25.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:25:25.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:25:25.271 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:25.277+0000 7fc9a418a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:25.273 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:25.277+0000 7fc9a418a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:25.274 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:25.281+0000 7fc9a418a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:25:25.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:25:25.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:25:25.641 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:25:25.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:25:25.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:25:25.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:25:25.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:25.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:25:25.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:25.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:25.825 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:25.829+0000 7fc9a418a7c0 -1 Falling back to public interface 2024-04-25T07:25:26.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:27.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:27.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:27.048 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:25:27.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:25:27.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:27.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:27.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:28.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:28.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:28.463 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:25:28.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:25:28.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:28.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:28.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:25:29.401 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:25:29.405+0000 7fc9a418a7c0 -1 osd.2 29 log_to_monitors true 2024-04-25T07:25:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:25:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:25:29.923 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:25:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:25:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:25:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:25:30.424 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/2786740163,v1:127.0.0.1:6811/2786740163] [v2:127.0.0.1:6812/2786740163,v1:127.0.0.1:6813/2786740163] exists,up a22a9735-1827-49a9-ad2a-a2ffcbe059d8 2024-04-25T07:25:30.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:25:30.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:25:30.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:25:30.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:30.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2024-04-25T07:25:30.980 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already in. 2024-04-25T07:25:30.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:25:30.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2024-04-25T07:25:31.527 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already in. 2024-04-25T07:25:31.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2024-04-25T07:25:36.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2024-04-25T07:25:36.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2024-04-25T07:25:36.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2024-04-25T07:25:36.548 INFO:tasks.workunit.client.0.smithi134.stdout:writing non-divergent object existing_4 2024-04-25T07:25:36.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:25:36.959 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:25:36.959 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 100 0 100 0 0 61700 0 0 16 84 16 active+undersized+degraded 2024-04-25T07:25:32.366263+0000 29'100 43:125 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:25:10.574551+0000 0 0 2024-04-25T07:25:36.959 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:25:36.959 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:25:36.959 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:25:36.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2024-04-25T07:25:37.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:25:37.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2024-04-25T07:25:37.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:25:37.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:25:37.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:25:37.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:25:37.092 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:25:37.092 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:25:37.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:25:37.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:25:37.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:25:37.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:25:37.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:25:37.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:25:37.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:25:37.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:25:37.162 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:25:37.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:25:37.607 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:25:37.607 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:25:37.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:25:37.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:25:37.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:25:37.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789958 2024-04-25T07:25:37.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789958 2024-04-25T07:25:37.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789958' 2024-04-25T07:25:37.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:25:37.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:25:37.904 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:25:37.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:25:37.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:25:37.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:25:37.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:25:37.907 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:25:38.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=167503724548 2024-04-25T07:25:38.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 167503724548 2024-04-25T07:25:38.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-158913789958 2-167503724548' 2024-04-25T07:25:38.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:25:38.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-158913789958 2024-04-25T07:25:38.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:25:38.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:25:38.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-158913789958 2024-04-25T07:25:38.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:25:38.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789958 2024-04-25T07:25:38.066 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 158913789958 2024-04-25T07:25:38.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 158913789958' 2024-04-25T07:25:38.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:25:38.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789958 -lt 158913789958 2024-04-25T07:25:38.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:25:38.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-167503724548 2024-04-25T07:25:38.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:25:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:25:38.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-167503724548 2024-04-25T07:25:38.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:25:38.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=167503724548 2024-04-25T07:25:38.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 167503724548' 2024-04-25T07:25:38.475 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 167503724548 2024-04-25T07:25:38.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:25:38.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 167503724546 -lt 167503724548 2024-04-25T07:25:38.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:25:39.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:25:39.905 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:25:40.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 167503724548 -lt 167503724548 2024-04-25T07:25:40.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:25:40.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:40.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:40.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:25:40.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:40.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:40.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:40.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:40.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:40.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:40.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:41.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:41.381 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:41.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:41.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:41.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:41.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:25:41.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:25:41.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:25:41.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:41.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:25:42.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:42.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:42.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:42.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:42.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:42.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:42.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:42.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:42.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:42.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:42.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:42.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:43.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:43.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:43.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:43.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:43.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:43.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:43.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:43.058 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:43.058 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:25:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:25:43.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:43.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:43.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:43.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:43.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:43.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:43.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:43.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:44.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:44.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:44.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:44.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:44.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:44.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:44.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:44.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:44.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:44.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:44.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:44.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:44.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:45.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:45.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:45.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:25:45.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:45.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:25:45.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:45.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:45.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:45.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:45.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:45.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:45.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:45.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:46.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:46.248 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:46.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:46.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:46.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:46.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:46.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:46.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:46.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:46.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:46.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:46.806 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:46.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:47.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:47.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:47.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:25:47.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:47.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:25:48.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:48.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:48.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:48.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:48.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:48.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:48.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:48.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:48.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:48.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:48.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:48.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:49.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:49.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:49.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:49.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:49.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:49.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:49.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:49.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:49.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:49.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:49.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:49.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:25:49.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:49.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:25:51.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:51.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:51.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:51.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:51.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:51.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:51.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:51.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:51.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:51.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:51.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:51.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:52.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:52.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:52.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:52.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:52.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:52.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:52.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:52.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:52.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:52.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:52.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:52.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:25:52.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:52.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:25:55.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:25:55.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:25:55.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:25:55.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:25:55.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:25:55.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:25:55.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:25:55.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:25:56.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:25:56.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:25:56.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:25:56.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:25:56.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:25:56.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:25:56.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:25:56.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:25:56.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:25:56.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:25:56.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:25:56.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:25:56.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:25:57.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:25:57.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:25:57.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:25:57.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:25:57.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:26:03.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:03.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:03.995 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:03.995 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:03.995 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:03.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:03.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:03.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:04.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:04.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:04.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:04.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:04.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:04.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:04.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:04.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:04.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:04.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:04.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:04.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:04.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:05.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:05.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:05.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:26:05.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:05.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:26:12.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:12.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:12.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:12.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:12.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:12.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:12.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:12.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:13.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:13.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:13.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:13.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:13.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:13.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:13.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:13.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:13.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:13.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:13.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:13.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:13.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:14.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:14.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:14.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:26:14.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:26:14.950 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_fingerprint": "376b955a-a0f5-4cb9-b610-d188083ed213", 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "timestamp": "2024-04-25T07:26:14.954267+0000", 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "tag": "", 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "health": { 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "status": "HEALTH_WARN", 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "checks": { 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 17 2024-04-25T07:26:14.951 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [], 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "OSD_DOWN": { 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 osds down", 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "osd.1 (root=default,host=smithi134) is down" 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.952 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: "PG_DEGRADED": { 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.953 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:26:14.954 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "mutes": [] 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_first_committed": 1, 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_last_committed": 1, 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap": { 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:22:19.571082Z", 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:22:19.571082Z", 2024-04-25T07:26:14.955 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release": 19, 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release_name": "squid", 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "election_strategy": 1, 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "disallowed_leaders: ": "", 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": false, 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "tiebreaker_mon": "", 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_ranks: ": "", 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "features": { 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "persistent": [ 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "kraken", 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "luminous", 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "mimic", 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap-prune", 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "nautilus", 2024-04-25T07:26:14.956 INFO:tasks.workunit.client.0.smithi134.stdout: "octopus", 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "pacific", 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "elector-pinging", 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "quincy", 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "reef", 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "squid" 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "optional": [] 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "mons": [ 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "rank": 0, 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "a", 2024-04-25T07:26:14.957 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 0 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 0, 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 0, 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_location": "{}" 2024-04-25T07:26:14.958 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "quorum": [ 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: 0 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap": { 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 43, 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:22:20.591529+0000", 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:25:32.038657+0000", 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "last_up_change": "2024-04-25T07:25:29.693715+0000", 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "last_in_change": "2024-04-25T07:25:31.375211+0000", 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_num": 5800384, 2024-04-25T07:26:14.959 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_set": [ 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "nodown", 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "noin", 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "noout", 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "pglog_hardlimit", 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snapdirs", 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "recovery_deletes", 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "sortbitwise" 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_version": 10, 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_snapshot": "", 2024-04-25T07:26:14.960 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_max": 1, 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "max_osd": 3, 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "min_compat_client": "jewel", 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "require_osd_release": "squid", 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "allow_crimson": false, 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "pools": [ 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_name": "test", 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "create_time": "2024-04-25T07:24:16.913964+0000", 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": 16389, 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 3, 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 3, 2024-04-25T07:26:14.961 INFO:tasks.workunit.client.0.smithi134.stdout: "min_size": 2, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_rule": 1, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "object_hash": 2, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num": 1, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num": 1, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num_target": 1, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_target": 1, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_pending": 1, 2024-04-25T07:26:14.962 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pg_merge_meta": { 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "source_pgid": "0.0", 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "ready_epoch": 0, 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_started": 0, 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": 0, 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "source_version": "0'0", 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "target_version": "0'0" 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "last_change": "30", 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend": "0", 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "auid": 0, 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_mode": "pool", 2024-04-25T07:26:14.963 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_seq": 1, 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_epoch": 30, 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_snaps": [ 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "snapid": 1, 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "stamp": "2024-04-25T07:25:05.571647+0000", 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap1" 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps": "[]", 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_bytes": 0, 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_objects": 0, 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "tiers": [], 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "tier_of": -1, 2024-04-25T07:26:14.964 INFO:tasks.workunit.client.0.smithi134.stdout: "read_tier": -1, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "write_tier": -1, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_mode": "none", 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_bytes": 0, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_objects": 0, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_flush_age": 0, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_evict_age": 0, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profile": "myprofile", 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_params": { 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "none" 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_period": 0, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_count": 0, 2024-04-25T07:26:14.965 INFO:tasks.workunit.client.0.smithi134.stdout: "use_gmt_hitset": true, 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "grade_table": [], 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "stripe_width": 8192, 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "expected_num_objects": 0, 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "fast_read": false, 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "options": {}, 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "application_metadata": {} 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: "osds": [ 2024-04-25T07:26:14.966 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "c09e27c0-df59-4ace-bcfa-29969022462f", 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 5, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 29, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 37, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 41, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 31, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:26:14.967 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2626715406 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2626715406 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.968 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2626715406 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2626715406 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.969 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2626715406 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2626715406 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.970 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2626715406 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2626715406 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6803/2626715406", 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6805/2626715406", 2024-04-25T07:26:14.971 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6809/2626715406", 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6807/2626715406", 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "87b81a46-8982-455b-8e0b-0fac08f51bc8", 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 0, 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T07:26:14.972 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 10, 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 25, 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 32, 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:26:14.973 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:26:14.974 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:14.975 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.976 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/3821439290", 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/3821439290", 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3821439290", 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3821439290", 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "exists" 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "a22a9735-1827-49a9-ad2a-a2ffcbe059d8", 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:26:14.977 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 15, 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 29, 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 39, 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 0, 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 33, 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2786740163 2024-04-25T07:26:14.978 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2786740163 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2786740163 2024-04-25T07:26:14.979 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2786740163 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2786740163 2024-04-25T07:26:14.980 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2786740163 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2786740163 2024-04-25T07:26:14.981 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 2786740163 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/2786740163", 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/2786740163", 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/2786740163", 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/2786740163", 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:26:14.982 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_xinfo": [ 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:25:16.920767+0000", 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:22:57.075581+0000", 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.983 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:25:17.545355+0000", 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:23:34.453911+0000", 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:25:18.143692+0000", 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:26:14.984 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:24:12.021478+0000", 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap": [], 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_items": [], 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_primaries": [], 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_temp": [], 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_temp": [], 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "blocklist": {}, 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "range_blocklist": {}, 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profiles": { 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "default": { 2024-04-25T07:26:14.985 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "2", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van" 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "myprofile": { 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-device-class": "", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-failure-domain": "osd", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-num-failure-domains": "0", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-osds-per-failure-domain": "0", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-root": "default", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "1", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T07:26:14.986 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van", 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "w": "8" 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps_queue": [], 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "new_removed_snaps": [], 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "new_purged_snaps": [], 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_node_flags": {}, 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "device_class_flags": {}, 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": { 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_enabled": false, 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_bucket_count": 0, 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:26:14.987 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_bucket": 0 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_metadata": [ 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6804/2626715406,v1:127.0.0.1:6805/2626715406]", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:26:14.988 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:22:27.515190Z", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:26:14.989 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6802/2626715406,v1:127.0.0.1:6803/2626715406]", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/2626715406,v1:127.0.0.1:6809/2626715406]", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/2626715406,v1:127.0.0.1:6807/2626715406]", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:26:14.990 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:26:14.991 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/3821439290,v1:127.0.0.1:6813/3821439290]", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:26:14.992 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:23:04.935875Z", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:26:14.993 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/3821439290,v1:127.0.0.1:6811/3821439290]", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3821439290,v1:127.0.0.1:6817/3821439290]", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3821439290,v1:127.0.0.1:6815/3821439290]", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:26:14.994 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/2786740163,v1:127.0.0.1:6813/2786740163]", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:26:14.995 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:14.996 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:23:41.867226Z", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/2786740163,v1:127.0.0.1:6811/2786740163]", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/2786740163,v1:127.0.0.1:6817/2786740163]", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/2786740163,v1:127.0.0.1:6815/2786740163]", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:26:14.997 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_clean_epochs": { 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "min_last_epoch_clean": 0, 2024-04-25T07:26:14.998 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": { 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: "per_pool": [] 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_epochs": [ 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 37 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 39 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_first_committed": 1, 2024-04-25T07:26:14.999 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_last_committed": 43, 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: "crushmap": { 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": [ 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.0", 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.1", 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.000 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.2", 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: "types": [ 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 0, 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd" 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "host" 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.001 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 2, 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "chassis" 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 3, 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "rack" 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 4, 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "row" 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.002 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 5, 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pdu" 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 6, 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pod" 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 7, 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "room" 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 8, 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "datacenter" 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.003 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 9, 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "zone" 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 10, 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "region" 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "root" 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: "buckets": [ 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.004 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -1, 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default", 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.005 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -2, 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default~hdd", 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.006 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134", 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.007 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134~hdd", 2024-04-25T07:26:15.008 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:15.009 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: "rules": [ 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 0, 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "replicated_rule", 2024-04-25T07:26:15.010 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_firstn", 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:15.011 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 1, 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "test", 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 3, 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "set_chooseleaf_tries", 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 5 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "set_choose_tries", 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 100 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.012 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_indep", 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:15.013 INFO:tasks.workunit.client.0.smithi134.stdout: "tunables": { 2024-04-25T07:26:15.014 INFO:tasks.workunit.client.0.smithi134.stderr:report 4008676019 2024-04-25T07:26:15.014 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_tries": 0, 2024-04-25T07:26:15.014 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:26:15.014 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_total_tries": 50, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_stable": 1, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_descents": 100, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_collision_tries": 100, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "straw_calc_version": 1, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "profile": "jewel", 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_tunables": 1, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "legacy_tunables": 0, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "minimum_required_version": "jewel", 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables": 1, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables2": 1, 2024-04-25T07:26:15.015 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v2_rules": 1, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables3": 1, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v3_rules": 0, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v4_buckets": 1, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables5": 1, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v5_rules": 0, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "has_msr_rules": 0 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_args": {} 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "fsmap": { 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "default_fscid": -1, 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": { 2024-04-25T07:26:15.016 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": {}, 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "ro_compat": {}, 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "incompat": { 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_1": "base v0.20", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_8": "no anchor table", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_9": "file layout v2", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_11": "minor log segments", 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.017 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_flags": { 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "enable_multiple": true, 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "ever_enabled_multiple": true 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "standbys": [], 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "filesystems": [] 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "auth": { 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 5, 2024-04-25T07:26:15.018 INFO:tasks.workunit.client.0.smithi134.stdout: "num_secrets": 3 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_active": 1, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_unknown": 0, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osd": 3, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_sum": { 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 62667, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 101, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 1, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 303, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 101, 2024-04-25T07:26:15.019 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 101, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 101, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 101, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:26:15.020 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:26:15.021 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:26:15.022 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 18, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 18, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 3, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 3, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 0 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum": { 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:26:15.023 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83668, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2164, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314489132, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036871168, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2215936, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 1550569, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:15.024 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:26:15.025 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum_by_class": { 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "hdd": { 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:26:15.026 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 83668, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2164, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 10, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 81461, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314489132, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322036871168, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2215936, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 1550569, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:26:15.027 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 10392, 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 83416936 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:26:15.028 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_stats": [ 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 62667, 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 101, 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 1, 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 303, 2024-04-25T07:26:15.029 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 101, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 101, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 101, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 101, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:26:15.030 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:26:15.031 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 1241088, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 1241088, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:26:15.032 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 18, 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 18, 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 3, 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 3, 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 3 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_stats": [ 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:26:15.033 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 158913789965 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 42949672997 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 167503724555 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_state": [ 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "active+undersized+degraded", 2024-04-25T07:26:15.034 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 1 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_osd": [ 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2147483647, 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:15.035 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 1, 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 0, 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": [ 2024-04-25T07:26:15.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": {} 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "servicemap": { 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 5, 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:25:14.001629+0000", 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "services": {} 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_last_committed": 106, 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_first_committed": 1, 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_last_committed": 411, 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "paxos": { 2024-04-25T07:26:15.037 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:26:15.038 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 565, 2024-04-25T07:26:15.038 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pn": 0, 2024-04-25T07:26:15.038 INFO:tasks.workunit.client.0.smithi134.stdout: "accepted_pn": 0 2024-04-25T07:26:15.038 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:15.038 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:26:15.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:26:15.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:26:15.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2024-04-25T07:26:15.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:15.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:15.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:15.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:15.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:15.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:15.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2024-04-25T07:26:15.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:26:15.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:26:15.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-04-25T07:26:15.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:26:15.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2024-04-25T07:26:15.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2024-04-25T07:26:16.507 INFO:tasks.workunit.client.0.smithi134.stderr:PG '1.0' not found 2024-04-25T07:26:16.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:26:16.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2024-04-25T07:26:16.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/0 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:16.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:16.801 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:16.801 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:16.801 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:26:16.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/0 2024-04-25T07:26:16.803 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:26:16.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:26:16.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:16.803 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/0/whoami 2024-04-25T07:26:16.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:26:16.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:26:16.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:26:16.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:26:16.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:16.847+0000 7f113f8a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:16.847 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:16.851+0000 7f113f8a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:16.849 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:16.851+0000 7f113f8a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:17.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:26:17.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:17.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:26:17.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:17.216 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:26:17.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:17.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:17.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:17.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:17.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:26:17.397 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:17.399+0000 7f113f8a87c0 -1 Falling back to public interface 2024-04-25T07:26:17.663 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 37 up_thru 41 down_at 31 last_clean_interval [5,29) [v2:127.0.0.1:6802/2626715406,v1:127.0.0.1:6803/2626715406] [v2:127.0.0.1:6804/2626715406,v1:127.0.0.1:6805/2626715406] exists,up c09e27c0-df59-4ace-bcfa-29969022462f 2024-04-25T07:26:17.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:17.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:17.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:17.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:26:17.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2024-04-25T07:26:17.665 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:26:17.665 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:26:17.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:26:17.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:26:17.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:26:17.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:26:17.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2024-04-25T07:26:17.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:26:17.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:26:17.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=2 2024-04-25T07:26:17.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:26:17.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2024-04-25T07:26:17.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2024-04-25T07:26:19.125 INFO:tasks.workunit.client.0.smithi134.stderr:PG '1.0' not found 2024-04-25T07:26:19.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:26:19.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2024-04-25T07:26:19.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:26:19.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:26:19.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:26:19.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/2 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:26:19.662 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:26:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:26:19.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:26:19.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:26:19.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/2 2024-04-25T07:26:19.665 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:26:19.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:26:19.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:26:19.666 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/2/whoami 2024-04-25T07:26:19.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:26:19.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:26:19.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:26:19.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:26:19.706 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:19.707+0000 7f2bdc0b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:19.709 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:19.711+0000 7f2bdc0b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:19.710 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:19.711+0000 7f2bdc0b57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:26:19.956 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:19.959+0000 7f113f8a87c0 -1 osd.0 43 log_to_monitors true 2024-04-25T07:26:20.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:26:20.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:26:20.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:26:20.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:26:20.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:26:20.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:26:20.176 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:26:20.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:26:20.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:26:20.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:26:20.270 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:20.271+0000 7f2bdc0b57c0 -1 Falling back to public interface 2024-04-25T07:26:20.641 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 33 last_clean_interval [15,29) [v2:127.0.0.1:6810/2786740163,v1:127.0.0.1:6811/2786740163] [v2:127.0.0.1:6812/2786740163,v1:127.0.0.1:6813/2786740163] exists,up a22a9735-1827-49a9-ad2a-a2ffcbe059d8 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/divergent-priors 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=1 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2024-04-25T07:26:20.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2024-04-25T07:26:22.221 INFO:tasks.workunit.client.0.smithi134.stderr:PG '1.0' not found 2024-04-25T07:26:22.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-04-25T07:26:22.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T07:26:22.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2024-04-25T07:26:22.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:26:22.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:26:22.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:26:22.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T07:26:22.513 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:26:22.513 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:26:22.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:26:22.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:26:22.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:26:22.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T07:26:22.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:26:22.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:26:22.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:26:22.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:26:22.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:26:22.901 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:26:22.903+0000 7f2bdc0b57c0 -1 osd.2 43 log_to_monitors true 2024-04-25T07:26:23.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:26:23.094 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:26:23.094 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:26:23.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:26:23.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:26:23.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:26:23.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=197568495619 2024-04-25T07:26:23.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 197568495619 2024-04-25T07:26:23.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-197568495619' 2024-04-25T07:26:23.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:26:23.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:26:23.490 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-04-25T07:26:23.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T07:26:23.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T07:26:23.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T07:26:23.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:26:23.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:26:23.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=3 2024-04-25T07:26:23.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 3 2024-04-25T07:26:23.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-197568495619 2-3' 2024-04-25T07:26:23.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:26:23.662 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-197568495619 2024-04-25T07:26:23.662 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:26:23.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:26:23.664 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-197568495619 2024-04-25T07:26:23.664 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:26:23.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=197568495619 2024-04-25T07:26:23.665 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 197568495619 2024-04-25T07:26:23.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 197568495619' 2024-04-25T07:26:23.665 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:26:24.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 197568495619 -lt 197568495619 2024-04-25T07:26:24.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:26:24.190 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:26:24.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-3 2024-04-25T07:26:24.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:26:24.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-3 2024-04-25T07:26:24.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:26:24.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=3 2024-04-25T07:26:24.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 3' 2024-04-25T07:26:24.195 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 3 2024-04-25T07:26:24.195 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:26:24.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 3 -lt 3 2024-04-25T07:26:24.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:26:24.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:24.701 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:25.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:26:25.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:25.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:25.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:25.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:25.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:25.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:25.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:25.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:25.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:25.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:25.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:26.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:26.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T07:26:26.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:26:26.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T07:26:26.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:26.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:26:26.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:26.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:26.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:26.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:26.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:26.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:26.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:26.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:26.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:26.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:26.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:26.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:27.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:27.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:27.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:27.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:27.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:27.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:27.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:27.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:27.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:27.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:27.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:27.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 8 )) 2024-04-25T07:26:27.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:27.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T07:26:28.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:28.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:28.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:28.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:28.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:28.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:28.050 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:28.050 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:28.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:28.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:28.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:28.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:29.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:29.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:29.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:29.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:29.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:29.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:29.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:29.042 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:29.042 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:29.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:29.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:29.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 8 )) 2024-04-25T07:26:29.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:29.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T07:26:29.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:29.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:29.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:29.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:29.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:29.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:29.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:29.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:30.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:30.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:30.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:30.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:30.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:30.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:30.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:31.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:31.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:31.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 8 )) 2024-04-25T07:26:31.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:31.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T07:26:32.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:32.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:32.310 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:32.310 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:32.310 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:32.310 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:32.310 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:32.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:32.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:32.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:32.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:32.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:33.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:33.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:33.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:33.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:33.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:33.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:33.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:33.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:33.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:33.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:33.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:33.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 8 )) 2024-04-25T07:26:33.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:33.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T07:26:35.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:35.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:35.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:35.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:35.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:35.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:35.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:35.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:35.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:35.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:35.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:35.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:36.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:36.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:36.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:36.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:36.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:36.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:36.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:36.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:36.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:36.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:36.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:36.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 8 )) 2024-04-25T07:26:36.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:36.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T07:26:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:40.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:40.173 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:40.173 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:40.173 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:40.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:40.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:40.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:40.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:40.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:40.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:41.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:41.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:41.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:41.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:41.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:41.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:41.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:41.115 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:41.115 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:41.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:41.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:41.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 8 )) 2024-04-25T07:26:41.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:41.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T07:26:48.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:48.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:48.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:48.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:48.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:48.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:48.075 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:48.075 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:48.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:48.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:48.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:48.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:49.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:49.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:49.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:49.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:49.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:49.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:49.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:49.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:49.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:49.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:49.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:49.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 8 )) 2024-04-25T07:26:49.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:26:49.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 7.3 2024-04-25T07:26:56.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:26:56.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:26:56.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:26:56.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:26:56.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:26:56.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:26:56.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:26:56.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:26:57.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T07:26:57.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:26:57.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:26:57.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:26:57.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T07:26:57.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T07:26:57.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T07:26:57.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T07:26:57.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T07:26:57.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T07:26:57.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T07:26:57.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T07:26:57.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T07:26:58.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T07:26:58.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T07:26:58.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 8 )) 2024-04-25T07:26:58.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T07:26:58.986 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:26:58.986 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_fingerprint": "376b955a-a0f5-4cb9-b610-d188083ed213", 2024-04-25T07:26:58.986 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:58.986 INFO:tasks.workunit.client.0.smithi134.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T07:26:58.986 INFO:tasks.workunit.client.0.smithi134.stdout: "timestamp": "2024-04-25T07:26:58.989522+0000", 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "tag": "", 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "health": { 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "status": "HEALTH_WARN", 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "checks": { 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "OSDMAP_FLAGS": { 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "nodown,noout,noin flag(s) set", 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 17 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [], 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "OSD_DOWN": { 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:58.987 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 osds down", 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "osd.1 (root=default,host=smithi134) is down" 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: "PG_DEGRADED": { 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:26:58.988 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [NONE,0,2]" 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 pool(s) do not have an application enabled", 2024-04-25T07:26:58.989 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "application not enabled on pool 'test'" 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: "mutes": [] 2024-04-25T07:26:58.990 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_first_committed": 1, 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_last_committed": 1, 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap": { 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:22:19.571082Z", 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:22:19.571082Z", 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release": 19, 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release_name": "squid", 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "election_strategy": 1, 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "disallowed_leaders: ": "", 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": false, 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "tiebreaker_mon": "", 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_ranks: ": "", 2024-04-25T07:26:58.991 INFO:tasks.workunit.client.0.smithi134.stdout: "features": { 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "persistent": [ 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "kraken", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "luminous", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "mimic", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap-prune", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "nautilus", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "octopus", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "pacific", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "elector-pinging", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "quincy", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "reef", 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "squid" 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:58.992 INFO:tasks.workunit.client.0.smithi134.stdout: "optional": [] 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: "mons": [ 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: "rank": 0, 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "a", 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115", 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 0 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:58.993 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7115/0", 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:7115/0", 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 0, 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 0, 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_location": "{}" 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: "quorum": [ 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: 0 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap": { 2024-04-25T07:26:58.994 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 51, 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "09bc8ccf-5f5f-45db-a8f0-078cc9abddcf", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T07:22:20.591529+0000", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:26:23.744173+0000", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "last_up_change": "2024-04-25T07:26:23.364917+0000", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "last_in_change": "2024-04-25T07:25:31.375211+0000", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_num": 5800384, 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_set": [ 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "nodown", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "noin", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "noout", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "pglog_hardlimit", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snapdirs", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "recovery_deletes", 2024-04-25T07:26:58.995 INFO:tasks.workunit.client.0.smithi134.stdout: "sortbitwise" 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_version": 12, 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_snapshot": "", 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_max": 1, 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "max_osd": 3, 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "require_min_compat_client": "luminous", 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "min_compat_client": "jewel", 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "require_osd_release": "squid", 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "allow_crimson": false, 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: "pools": [ 2024-04-25T07:26:58.996 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_name": "test", 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "create_time": "2024-04-25T07:24:16.913964+0000", 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": 16389, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 3, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 3, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "min_size": 2, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_rule": 1, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "object_hash": 2, 2024-04-25T07:26:58.997 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_autoscale_mode": "off", 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num": 1, 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num": 1, 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num_target": 1, 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_target": 1, 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_pending": 1, 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pg_merge_meta": { 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "source_pgid": "0.0", 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "ready_epoch": 0, 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_started": 0, 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": 0, 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "source_version": "0'0", 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "target_version": "0'0" 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:58.998 INFO:tasks.workunit.client.0.smithi134.stdout: "last_change": "30", 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend": "0", 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "auid": 0, 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_mode": "pool", 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_seq": 1, 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_epoch": 30, 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_snaps": [ 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "snapid": 1, 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "stamp": "2024-04-25T07:25:05.571647+0000", 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "snap1" 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:58.999 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps": "[]", 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_bytes": 0, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_objects": 0, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "tiers": [], 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "tier_of": -1, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "read_tier": -1, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "write_tier": -1, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_mode": "none", 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_bytes": 0, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_objects": 0, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_flush_age": 0, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_evict_age": 0, 2024-04-25T07:26:59.000 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profile": "myprofile", 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_params": { 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "none" 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_period": 0, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_count": 0, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "use_gmt_hitset": true, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_search_last_n": 0, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "grade_table": [], 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "stripe_width": 8192, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "expected_num_objects": 0, 2024-04-25T07:26:59.001 INFO:tasks.workunit.client.0.smithi134.stdout: "fast_read": false, 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "options": {}, 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "application_metadata": {} 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "osds": [ 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "c09e27c0-df59-4ace-bcfa-29969022462f", 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 37, 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 43, 2024-04-25T07:26:59.002 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 46, 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 50, 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 45, 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6802", 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1375774847 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6803", 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1375774847 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.003 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6804", 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1375774847 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6805", 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1375774847 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.004 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6808", 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1375774847 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6809", 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1375774847 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:26:59.005 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6806", 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1375774847 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6807", 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1375774847 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6803/1375774847", 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6805/1375774847", 2024-04-25T07:26:59.006 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6809/1375774847", 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6807/1375774847", 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "87b81a46-8982-455b-8e0b-0fac08f51bc8", 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 0, 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T07:26:59.007 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 10, 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 25, 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 32, 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.008 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.009 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.010 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3821439290 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/3821439290", 2024-04-25T07:26:59.011 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/3821439290", 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/3821439290", 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/3821439290", 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "exists" 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "a22a9735-1827-49a9-ad2a-a2ffcbe059d8", 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 39, 2024-04-25T07:26:59.012 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 43, 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 50, 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 0, 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 49, 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 4096685795 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T07:26:59.013 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 4096685795 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 4096685795 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T07:26:59.014 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 4096685795 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 4096685795 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 4096685795 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.015 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 4096685795 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 4096685795 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.016 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/4096685795", 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/4096685795", 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/4096685795", 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/4096685795", 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_xinfo": [ 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:26:20.294519+0000", 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:26:59.017 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:22:57.075581+0000", 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:25:17.545355+0000", 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:23:34.453911+0000", 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:26:59.018 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "2024-04-25T07:26:23.217534+0000", 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T07:24:12.021478+0000", 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap": [], 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_items": [], 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_primaries": [], 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_temp": [], 2024-04-25T07:26:59.019 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_temp": [], 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "blocklist": {}, 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "range_blocklist": {}, 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profiles": { 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "default": { 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "2", 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van" 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "myprofile": { 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-device-class": "", 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-failure-domain": "osd", 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-num-failure-domains": "0", 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-osds-per-failure-domain": "0", 2024-04-25T07:26:59.020 INFO:tasks.workunit.client.0.smithi134.stdout: "crush-root": "default", 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "jerasure-per-chunk-alignment": "false", 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "1", 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van", 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "w": "8" 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps_queue": [], 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "new_removed_snaps": [], 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "new_purged_snaps": [], 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_node_flags": {}, 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "device_class_flags": {}, 2024-04-25T07:26:59.021 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": { 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_enabled": false, 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_bucket_count": 0, 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "degraded_stretch_mode": 0, 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "recovering_stretch_mode": 0, 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_bucket": 0 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_metadata": [ 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6804/1375774847,v1:127.0.0.1:6805/1375774847]", 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:26:59.022 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:26:59.023 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:22:27.515190Z", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6802/1375774847,v1:127.0.0.1:6803/1375774847]", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:26:59.024 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/1375774847,v1:127.0.0.1:6809/1375774847]", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/1375774847,v1:127.0.0.1:6807/1375774847]", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/0", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:26:59.025 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/3821439290,v1:127.0.0.1:6813/3821439290]", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:26:59.026 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:26:59.027 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:23:04.935875Z", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/3821439290,v1:127.0.0.1:6811/3821439290]", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/3821439290,v1:127.0.0.1:6817/3821439290]", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/3821439290,v1:127.0.0.1:6815/3821439290]", 2024-04-25T07:26:59.028 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/1", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.029 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/4096685795,v1:127.0.0.1:6813/4096685795]", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T07:26:59.030 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T07:23:41.867226Z", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T07:26:59.031 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/4096685795,v1:127.0.0.1:6811/4096685795]", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/4096685795,v1:127.0.0.1:6817/4096685795]", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/4096685795,v1:127.0.0.1:6815/4096685795]", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T07:26:59.032 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/divergent-priors/2", 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_clean_epochs": { 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: "min_last_epoch_clean": 26, 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": { 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: "per_pool": [ 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: "floor": 26 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.033 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_epochs": [ 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 46 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 50 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_first_committed": 1, 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_last_committed": 51, 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: "crushmap": { 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": [ 2024-04-25T07:26:59.034 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.0", 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.1", 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.2", 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.035 INFO:tasks.workunit.client.0.smithi134.stdout: "types": [ 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 0, 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd" 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "host" 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 2, 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "chassis" 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 3, 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "rack" 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 4, 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "row" 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 5, 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pdu" 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 6, 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pod" 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 7, 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "room" 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.037 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 8, 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "datacenter" 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 9, 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "zone" 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 10, 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "region" 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "root" 2024-04-25T07:26:59.038 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "buckets": [ 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -1, 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default", 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.039 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -2, 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default~hdd", 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.040 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134", 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.041 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134~hdd", 2024-04-25T07:26:59.042 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 19206, 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.043 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "rules": [ 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 0, 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "replicated_rule", 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T07:26:59.044 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_firstn", 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 1, 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "test", 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 3, 2024-04-25T07:26:59.045 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "set_chooseleaf_tries", 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 5 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "set_choose_tries", 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 100 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.046 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.047 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_indep", 2024-04-25T07:26:59.047 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T07:26:59.047 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T07:26:59.047 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.047 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.047 INFO:tasks.workunit.client.0.smithi134.stderr:report 2090371360 2024-04-25T07:26:59.047 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "tunables": { 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_tries": 0, 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_fallback_tries": 0, 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_total_tries": 50, 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_descend_once": 1, 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_vary_r": 1, 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_stable": 1, 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_descents": 100, 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_collision_tries": 100, 2024-04-25T07:26:59.048 INFO:tasks.workunit.client.0.smithi134.stdout: "straw_calc_version": 1, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "allowed_bucket_algs": 54, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "profile": "jewel", 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_tunables": 1, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "legacy_tunables": 0, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "minimum_required_version": "jewel", 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables": 1, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables2": 1, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v2_rules": 1, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables3": 1, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v3_rules": 0, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v4_buckets": 1, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables5": 1, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v5_rules": 0, 2024-04-25T07:26:59.049 INFO:tasks.workunit.client.0.smithi134.stdout: "has_msr_rules": 0 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_args": {} 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "fsmap": { 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "default_fscid": -1, 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": { 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": {}, 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "ro_compat": {}, 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "incompat": { 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_1": "base v0.20", 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_2": "client writeable ranges", 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T07:26:59.050 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_4": "dir inode in separate object", 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_8": "no anchor table", 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_9": "file layout v2", 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_10": "snaprealm v2", 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_11": "minor log segments", 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_flags": { 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "enable_multiple": true, 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "ever_enabled_multiple": true 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "standbys": [], 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: "filesystems": [] 2024-04-25T07:26:59.051 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_first_committed": 1, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_last_committed": 1, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "auth": { 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 5, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "num_secrets": 3 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_active": 1, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_unknown": 0, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osd": 3, 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_sum": { 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:26:59.052 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 62667, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 101, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 1, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 303, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 101, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 101, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 101, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 101, 2024-04-25T07:26:59.053 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:26:59.054 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 0, 2024-04-25T07:26:59.055 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 0, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 18, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 18, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 3, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 3, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 0 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum": { 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:26:59.056 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 85096, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2292, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 15, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 82736, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314487704, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322035408896, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2347008, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 1605477, 2024-04-25T07:26:59.057 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 16257, 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 84721791 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:26:59.058 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum_by_class": { 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "hdd": { 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 3, 2024-04-25T07:26:59.059 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 3, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 3, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 3, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 314572800, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 85096, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 2292, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 15, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 82736, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 314487704, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 322122547200, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 322035408896, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 2347008, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 1605477, 2024-04-25T07:26:59.060 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 16257, 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 84721791 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 0, 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [], 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 1 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T07:26:59.061 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_stats": [ 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 62667, 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 101, 2024-04-25T07:26:59.062 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 1, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 303, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 101, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 101, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 101, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 101, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T07:26:59.063 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T07:26:59.064 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 1241088, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 1241088, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T07:26:59.065 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 18, 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 18, 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 3, 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 3, 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 3 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_stats": [ 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 197568495626 2024-04-25T07:26:59.066 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 42949672997 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 214748364809 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_state": [ 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "active+undersized+degraded", 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 1 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.067 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_osd": [ 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2147483647, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 1, 2024-04-25T07:26:59.068 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 1, 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 0, 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 0, 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": [ 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": {} 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.069 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "servicemap": { 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 5, 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T07:25:14.001629+0000", 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "services": {} 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_first_committed": 1, 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_last_committed": 128, 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_first_committed": 1, 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_last_committed": 491, 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "paxos": { 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 674, 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pn": 0, 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: "accepted_pn": 0 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T07:26:59.070 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:26:59.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T07:26:59.071 INFO:tasks.workunit.client.0.smithi134.stdout:delay recovery 2024-04-25T07:26:59.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2024-04-25T07:26:59.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:26:59.414 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:26:59.414 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:26:24.989339+0000 43'102 51:138 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:26:59.414 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:26:59.414 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:26:59.415 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:26:59.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:26:59.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2024-04-25T07:26:59.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:26:59.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:26:59.431 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:59.431 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:59.431 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:26:59.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.0.asok 2024-04-25T07:26:59.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:26:59.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.0.asok set_recovery_delay 100000 2024-04-25T07:26:59.540 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:26:59.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2024-04-25T07:26:59.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2024-04-25T07:26:59.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T07:26:59.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T07:26:59.555 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:26:59.555 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:26:59.555 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:26:59.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.2.asok 2024-04-25T07:26:59.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:26:59.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.2.asok set_recovery_delay 100000 2024-04-25T07:26:59.663 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:26:59.677 INFO:tasks.workunit.client.0.smithi134.stdout:revive divergent 1 2024-04-25T07:26:59.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2024-04-25T07:26:59.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:27:00.090 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:27:00.090 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:26:24.989339+0000 43'102 51:138 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:27:00.090 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:27:00.090 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:27:00.091 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:27:00.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2024-04-25T07:27:00.792 INFO:tasks.workunit.client.0.smithi134.stderr:noup is set 2024-04-25T07:27:00.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-04-25T07:27:00.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:27:00.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:27:00.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:00.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:00.818 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:00.818 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:00.818 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:27:00.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:27:00.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:27:00.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:27:00.820 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:27:00.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:27:00.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:27:00.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:27:00.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:27:00.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:27:00.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:27:00.869 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:00.873+0000 7f972741d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:00.871 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:00.873+0000 7f972741d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:00.872 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:00.877+0000 7f972741d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:01.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:880: activate_osd: return 0 2024-04-25T07:27:01.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2024-04-25T07:27:01.410 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:01.413+0000 7f972741d7c0 -1 Falling back to public interface 2024-04-25T07:27:04.033 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:04.037+0000 7f972741d7c0 -1 osd.1 30 log_to_monitors true 2024-04-25T07:27:06.299 INFO:tasks.workunit.client.0.smithi134.stdout:delay recovery divergent 2024-04-25T07:27:06.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2024-04-25T07:27:06.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:27:06.700 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:27:06.701 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:26:24.989339+0000 43'102 54:144 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:44:22.349659+0000 0 0 2024-04-25T07:27:06.701 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:27:06.701 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:27:06.701 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:27:06.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2024-04-25T07:27:06.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T07:27:06.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T07:27:06.718 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:06.718 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:06.718 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:27:06.718 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.26105/ceph-osd.1.asok 2024-04-25T07:27:06.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2024-04-25T07:27:06.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.26105/ceph-osd.1.asok set_recovery_delay 100000 2024-04-25T07:27:06.850 INFO:tasks.workunit.client.0.smithi134.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2024-04-25T07:27:06.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2024-04-25T07:27:07.485 INFO:tasks.workunit.client.0.smithi134.stderr:noup is unset 2024-04-25T07:27:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2024-04-25T07:27:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:27:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:27:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:27:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:27:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:27:07.505 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:27:07.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:07.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:08.023 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 46 up_thru 50 down_at 45 last_clean_interval [37,43) [v2:127.0.0.1:6802/1375774847,v1:127.0.0.1:6803/1375774847] [v2:127.0.0.1:6804/1375774847,v1:127.0.0.1:6805/1375774847] exists,up c09e27c0-df59-4ace-bcfa-29969022462f 2024-04-25T07:27:08.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:27:08.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:27:08.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:27:08.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2024-04-25T07:27:08.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:27:08.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:27:08.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:27:08.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:27:08.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:08.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:27:08.025 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:27:08.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:08.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:08.584 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/462670885,v1:127.0.0.1:6819/462670885] [v2:127.0.0.1:6820/462670885,v1:127.0.0.1:6821/462670885] exists,up 87b81a46-8982-455b-8e0b-0fac08f51bc8 2024-04-25T07:27:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:27:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:27:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:27:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2024-04-25T07:27:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:27:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:27:08.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:27:08.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:27:08.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:08.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:27:08.586 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:27:08.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:08.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:27:09.104 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 49 last_clean_interval [39,43) [v2:127.0.0.1:6810/4096685795,v1:127.0.0.1:6811/4096685795] [v2:127.0.0.1:6812/4096685795,v1:127.0.0.1:6813/4096685795] exists,up a22a9735-1827-49a9-ad2a-a2ffcbe059d8 2024-04-25T07:27:09.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:27:09.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:27:09.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:27:09.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:27:09.514 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:27:09.514 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:26:24.989339+0000 43'102 54:144 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:44:22.349659+0000 0 0 2024-04-25T07:27:09.514 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:27:09.514 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:27:09.515 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:27:09.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2024-04-25T07:27:09.529 INFO:tasks.workunit.client.0.smithi134.stdout:wait for peering 2024-04-25T07:27:09.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:27:09.937 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:27:09.937 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 0 101 0 0 62667 0 0 18 84 18 active+undersized+degraded 2024-04-25T07:26:24.989339+0000 43'102 54:144 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:44:22.349659+0000 0 0 2024-04-25T07:27:09.937 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:27:09.937 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:27:09.937 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:27:09.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2024-04-25T07:27:10.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2024-04-25T07:27:10.105 INFO:tasks.workunit.client.0.smithi134.stdout:killing divergent 1 2024-04-25T07:27:10.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:27:10.513 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:27:10.513 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-04-25T07:27:08.449877+0000 43'102 58:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:27:10.513 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:27:10.513 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:27:10.514 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:27:10.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2024-04-25T07:27:10.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:10.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:10.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:10.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:10.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:10.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:10.638 INFO:tasks.workunit.client.0.smithi134.stdout:reviving divergent 1 2024-04-25T07:27:10.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2024-04-25T07:27:10.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:27:11.046 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:27:11.046 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 101 4 4 0 0 62667 0 0 12 90 12 active+recovering+degraded 2024-04-25T07:27:08.449877+0000 43'102 58:133 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 no scrub is scheduled 0 0 2024-04-25T07:27:11.046 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:27:11.046 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:27:11.046 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:27:11.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/divergent-priors 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/divergent-priors/1 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:27:11.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:27:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:27:11.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:27:11.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:27:11.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:27:11.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/divergent-priors/1 2024-04-25T07:27:11.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:27:11.068 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:27:11.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=09bc8ccf-5f5f-45db-a8f0-078cc9abddcf --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.26105/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:27:11.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/divergent-priors/1/whoami 2024-04-25T07:27:11.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:27:11.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:27:11.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:27:11.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:27:11.099 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:11.101+0000 7f5e7099e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:11.101 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:11.105+0000 7f5e7099e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:11.103 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:11.105+0000 7f5e7099e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:11.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:27:11.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:27:11.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:27:11.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:27:11.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:27:11.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:11.465 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:27:11.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:27:11.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:11.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:27:11.650 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:11.653+0000 7f5e7099e7c0 -1 Falling back to public interface 2024-04-25T07:27:11.898 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 57 up_thru 57 down_at 32 last_clean_interval [10,30) [v2:127.0.0.1:6818/462670885,v1:127.0.0.1:6819/462670885] [v2:127.0.0.1:6820/462670885,v1:127.0.0.1:6821/462670885] exists,up 87b81a46-8982-455b-8e0b-0fac08f51bc8 2024-04-25T07:27:11.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:27:11.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:27:11.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:27:11.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2024-04-25T07:27:14.699 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:14.701+0000 7f5e7099e7c0 -1 osd.1 58 log_to_monitors true 2024-04-25T07:27:31.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2024-04-25T07:27:31.900 INFO:tasks.workunit.client.0.smithi134.stdout:allowing recovery 2024-04-25T07:27:31.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:27:32.297 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:27:32.297 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-04-25T07:27:17.231739+0000 58'103 63:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:56:57.770237+0000 0 0 2024-04-25T07:27:32.297 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:27:32.297 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:27:32.298 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:27:32.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:27:32.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T07:27:32.465 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:27:32.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:27:32.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T07:27:32.661 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:27:32.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2024-04-25T07:27:32.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T07:27:32.829 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T07:27:32.843 INFO:tasks.workunit.client.0.smithi134.stdout:reading divergent objects 2024-04-25T07:27:32.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2024-04-25T07:27:32.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2024-04-25T07:27:33.254 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T07:27:33.254 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 102 0 0 0 0 63284 0 0 13 90 13 active+clean 2024-04-25T07:27:17.231739+0000 58'103 63:157 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T07:24:16.981088+0000 0'0 2024-04-25T07:24:16.981088+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:56:57.770237+0000 0 0 2024-04-25T07:27:33.254 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T07:27:33.254 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T07:27:33.254 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T07:27:33.271 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2024-04-25T07:27:33.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2024-04-25T07:27:33.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:27:33.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2024-04-25T07:27:33.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:27:33.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2024-04-25T07:27:33.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:27:33.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2024-04-25T07:27:33.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2024-04-25T07:27:33.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2024-04-25T07:27:33.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2024-04-25T07:27:33.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-04-25T07:27:33.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-04-25T07:24:57.518945+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 20 _merge_object_divergent_entries: keeping 30'103 (29'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2024-04-25T07:24:57.518945+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 29'99 first_divergent_update: 30'103 last_divergent_update: 30'103 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-04-25T07:25:05.872312+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 20 _merge_object_divergent_entries: keeping 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-04-25T07:25:05.872312+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 29'99 first_divergent_update: 30'104 last_divergent_update: 30'104 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:27:33.442 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 30'104 2024-04-25T07:27:33.443 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 30'104 (29'99) modify 1:7a80c0d5:::existing_99:head by client.4655.0:1 2024-04-25T07:25:05.872312+0000 0 ObjectCleanRegions clean_offsets: [(617, 18446744073709550998)], clean_omap: true, new_object: false 2024-04-25T07:27:33.443 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-04-25T07:25:04.566961+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false,29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-04-25T07:25:05.572252+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4646.0:1 2024-04-25T07:25:04.566961+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 20 _merge_object_divergent_entries: keeping 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-04-25T07:25:05.572252+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'102 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 30'104 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'102 (29'101) modify 1:c156033e:::existing_100:head by client.4652.0:1 2024-04-25T07:25:05.572252+0000 0 ObjectCleanRegions clean_offsets: [(967, 18446744073709550648)], clean_omap: true, new_object: false 2024-04-25T07:27:33.444 INFO:tasks.workunit.client.0.smithi134.stdout:td/divergent-priors/osd.1.log:2024-04-25T07:27:08.153+0000 7f970b3a7640 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2024-04-25T07:27:33.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2024-04-25T07:27:33.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2024-04-25T07:27:33.452 INFO:tasks.workunit.client.0.smithi134.stdout:success 2024-04-25T07:27:33.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2024-04-25T07:27:33.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2024-04-25T07:27:33.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T07:27:33.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T07:27:33.964 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T07:27:33.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2024-04-25T07:27:33.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:33.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:33.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:33.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:33.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:34.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:34.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2024-04-25T07:27:34.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:27:34.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:27:34.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:27:34.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:34.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:34.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:34.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:34.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:34.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:34.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:34.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:34.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:34.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:34.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:34.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:34.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:34.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:34.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:34.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:34.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:34.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:27:34.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:27:34.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:34.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:34.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:27:34.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26105 2024-04-25T07:27:34.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:34.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:34.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:27:34.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/divergent-priors 0 2024-04-25T07:27:34.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2024-04-25T07:27:34.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:27:34.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2024-04-25T07:27:34.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:34.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:34.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:34.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:34.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:34.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:34.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:34.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:34.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:34.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:34.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:34.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:34.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:34.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:34.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:34.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:34.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:34.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:27:34.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2024-04-25T07:27:34.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:34.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:34.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.26105 2024-04-25T07:27:34.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.26105 2024-04-25T07:27:34.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:34.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:34.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:27:34.137 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:27:34.137 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:27:34.192 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2024-04-25T07:27:34.193 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2024-04-25T07:27:34.244 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:27:34.250 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:27:34.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:27:34.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:27:34.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/ec-error-rollforward 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none ' 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2024-04-25T07:27:34.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2024-04-25T07:27:34.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2024-04-25T07:27:34.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2024-04-25T07:27:34.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2024-04-25T07:27:34.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2024-04-25T07:27:34.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:27:34.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2024-04-25T07:27:34.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2024-04-25T07:27:34.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:27:34.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:27:34.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:27:34.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:27:34.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:27:34.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:27:34.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:27:34.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:27:34.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:27:34.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:34.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:27:34.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:27:34.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:27:34.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:27:34.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:27:34.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:27:34.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:27:34.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:34.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:27:34.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83166 2024-04-25T07:27:34.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:27:34.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:27:34.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2024-04-25T07:27:34.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:27:34.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:34.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:27:34.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.83166 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:27:34.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:27:34.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2024-04-25T07:27:34.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2024-04-25T07:27:34.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:27:34.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:34.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:34.726 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:34.726 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:34.726 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:27:34.726 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:27:34.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:27:34.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:27:34.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:27:34.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:27:34.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:27:34.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:27:34.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:27:34.948 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:27:34.948 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:27:34.948 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:27:34.948 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:34.948 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:34.948 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:27:34.948 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83166/ceph-mon.a.asok 2024-04-25T07:27:34.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:27:34.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83166/ceph-mon.a.asok config get fsid 2024-04-25T07:27:35.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:27:35.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:27:35.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:27:35.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:27:35.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:27:35.064 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:27:35.064 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:27:35.064 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:27:35.064 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:27:35.064 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:35.065 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:27:35.065 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.83166/ceph-mon.a.asok 2024-04-25T07:27:35.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:27:35.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.83166/ceph-mon.a.asok config get mon_host 2024-04-25T07:27:35.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2024-04-25T07:27:35.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ec-error-rollforward 2024-04-25T07:27:35.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:27:35.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:27:35.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:27:35.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ec-error-rollforward/x 2024-04-25T07:27:35.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:27:36.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:27:36.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:36.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:36.064 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:36.064 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:36.064 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:27:36.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:27:36.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:27:36.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/0 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-04-25T07:27:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:27:36.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/0 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e4f34214-daa6-46aa-b6a5-7ae719353e68 2024-04-25T07:27:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e4f34214-daa6-46aa-b6a5-7ae719353e68' 2024-04-25T07:27:36.102 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 e4f34214-daa6-46aa-b6a5-7ae719353e68 2024-04-25T07:27:36.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:27:36.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDoBSpm/I4tBxAA6OtirBNY9QCy4fwQmMlS1A== 2024-04-25T07:27:36.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDoBSpm/I4tBxAA6OtirBNY9QCy4fwQmMlS1A=="}' 2024-04-25T07:27:36.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e4f34214-daa6-46aa-b6a5-7ae719353e68 -i td/ec-error-rollforward/0/new.json 2024-04-25T07:27:36.415 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:27:36.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/0/new.json 2024-04-25T07:27:36.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDoBSpm/I4tBxAA6OtirBNY9QCy4fwQmMlS1A== --osd-uuid e4f34214-daa6-46aa-b6a5-7ae719353e68 2024-04-25T07:27:36.459 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:36.460+0000 7f90f76597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:36.493 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:36.492+0000 7f90f76597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:36.494 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:36.496+0000 7f90f76597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:36.494 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:36.496+0000 7f90f76597c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:27:36.494 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:36.496+0000 7f90f76597c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:27:36.494 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:36.496+0000 7f90f76597c0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2024-04-25T07:27:36.532 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:36.532+0000 7f90f76597c0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2024-04-25T07:27:42.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2024-04-25T07:27:42.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:27:42.806 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:27:42.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:27:42.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:27:43.363 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:27:43.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:27:43.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:27:43.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:27:43.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:27:43.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:27:43.392 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:43.396+0000 7f3aa2a597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:43.394 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:43.396+0000 7f3aa2a597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:43.396 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:43.396+0000 7f3aa2a597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:27:43.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:27:43.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:27:43.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:27:43.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:27:43.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:27:43.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:43.780 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:27:43.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:27:43.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:43.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:43.966 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:43.968+0000 7f3aa2a597c0 -1 Falling back to public interface 2024-04-25T07:27:44.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:45.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:45.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:45.254 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:27:45.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:27:45.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:45.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:45.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:46.613 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:27:46.616+0000 7f3aa2a597c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:27:46.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:46.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:46.662 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:27:46.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:27:46.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:46.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:47.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:48.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:48.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:48.285 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:27:48.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:27:48.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:48.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:48.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:49.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:49.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:49.760 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:27:49.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:27:49.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:49.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:50.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:51.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:51.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:51.235 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:27:51.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:27:51.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:51.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:51.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:52.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:52.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:52.880 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:27:52.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:27:52.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:52.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:53.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:54.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:54.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:54.316 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:27:54.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:27:54.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:54.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:54.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:55.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:55.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:55.783 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:27:55.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:27:55.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:55.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:56.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:57.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:57.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:57.212 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:27:57.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:27:57.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:57.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:57.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:27:58.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:27:58.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:27:58.743 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:27:58.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:27:58.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:27:58.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:27:59.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:00.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:00.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:00.222 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:28:00.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:28:00.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:00.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:01.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:02.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:02.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:02.179 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:28:02.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:28:02.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:02.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:03.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:04.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:04.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:04.009 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:28:04.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:28:04.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:04.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:04.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:05.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:05.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:05.914 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:28:05.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:28:05.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:05.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:06.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:07.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:07.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:07.626 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:28:07.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:28:07.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:07.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:08.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:09.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:09.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:09.499 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:28:09.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:28:09.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:09.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:10.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:11.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:11.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:11.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:28:11.345 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:28:11.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:11.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:28:11.833 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1546806672,v1:127.0.0.1:6803/1546806672] [v2:127.0.0.1:6804/1546806672,v1:127.0.0.1:6805/1546806672] exists,up e4f34214-daa6-46aa-b6a5-7ae719353e68 2024-04-25T07:28:11.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:28:11.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:28:11.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:28:11.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2024-04-25T07:28:11.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:28:11.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:28:11.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:28:11.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/1 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:28:11.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:28:11.838 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:28:11.838 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:11.838 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:28:11.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:28:11.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:28:11.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:28:11.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:28:11.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:28:11.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/1 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:28:11.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a3e29183-4e37-45a2-a26f-89aadbc8023b 2024-04-25T07:28:11.840 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 a3e29183-4e37-45a2-a26f-89aadbc8023b 2024-04-25T07:28:11.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a3e29183-4e37-45a2-a26f-89aadbc8023b' 2024-04-25T07:28:11.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:28:11.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALBipmpWuYMxAAgpZGUBAGpXhOcOViRUeJNQ== 2024-04-25T07:28:11.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALBipmpWuYMxAAgpZGUBAGpXhOcOViRUeJNQ=="}' 2024-04-25T07:28:11.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a3e29183-4e37-45a2-a26f-89aadbc8023b -i td/ec-error-rollforward/1/new.json 2024-04-25T07:28:12.344 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:28:12.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/1/new.json 2024-04-25T07:28:12.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALBipmpWuYMxAAgpZGUBAGpXhOcOViRUeJNQ== --osd-uuid a3e29183-4e37-45a2-a26f-89aadbc8023b 2024-04-25T07:28:12.393 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:12.395+0000 7f0ac1e3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:12.413 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:12.415+0000 7f0ac1e3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:12.414 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:12.415+0000 7f0ac1e3f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:12.415 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:12.415+0000 7f0ac1e3f7c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:28:12.415 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:12.415+0000 7f0ac1e3f7c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:28:12.415 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:12.415+0000 7f0ac1e3f7c0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2024-04-25T07:28:12.500 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:12.503+0000 7f0ac1e3f7c0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2024-04-25T07:28:18.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2024-04-25T07:28:18.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:28:18.879 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:28:18.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:28:18.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:28:19.451 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:28:19.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:28:19.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:28:19.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:28:19.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:28:19.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:28:19.480 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:19.483+0000 7f70b6eaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:19.483 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:19.483+0000 7f70b6eaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:19.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:19.483+0000 7f70b6eaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:19.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:28:19.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:28:19.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:28:19.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:28:19.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:28:19.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:19.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:28:19.885 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:28:19.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:19.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:20.042 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:20.043+0000 7f70b6eaf7c0 -1 Falling back to public interface 2024-04-25T07:28:20.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:21.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:21.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:21.322 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:28:21.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:28:21.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:21.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:21.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:22.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:22.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:22.726 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:28:22.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:28:22.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:22.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:22.911 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:22.911+0000 7f70b6eaf7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:28:23.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:24.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:24.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:24.151 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:28:24.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:28:24.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:24.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:24.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:25.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:25.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:25.594 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:28:25.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:28:25.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:25.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:26.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:27.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:27.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:27.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:28:27.186 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:28:27.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:27.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:27.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:28.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:28.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:28.689 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:28:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:28:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:29.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:30.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:30.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:30.221 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:28:30.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:28:30.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:30.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:30.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:31.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:31.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:31.757 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:28:31.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:28:31.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:31.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:32.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:33.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:33.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:33.268 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:28:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:28:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:33.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:34.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:34.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:34.820 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:28:34.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:28:34.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:34.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:35.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:36.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:36.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:36.263 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:28:36.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:28:36.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:36.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:36.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:37.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:37.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:37.789 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:28:37.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:28:37.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:37.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:38.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:39.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:39.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:39.558 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:28:39.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:28:39.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:39.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:40.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:41.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:41.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:41.232 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:28:41.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:28:41.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:41.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:42.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:43.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:43.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:43.508 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:28:43.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:28:43.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:43.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:44.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:45.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:45.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:45.408 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:28:45.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:28:45.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:45.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:46.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:47.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:47.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:47.255 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:28:47.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:28:47.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:47.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:48.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:49.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:49.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:49.474 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:28:49.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:28:49.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:49.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:28:49.890 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3713568849,v1:127.0.0.1:6811/3713568849] [v2:127.0.0.1:6812/3713568849,v1:127.0.0.1:6813/3713568849] exists,up a3e29183-4e37-45a2-a26f-89aadbc8023b 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/2 2024-04-25T07:28:49.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:28:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:28:49.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:28:49.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:28:49.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:28:49.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:28:49.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:28:49.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:28:49.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:28:49.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:28:49.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:28:49.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:28:49.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:28:49.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:28:49.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:28:49.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/2 2024-04-25T07:28:49.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:28:49.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13c11252-bb9a-403d-bf47-fa6cae3141cb 2024-04-25T07:28:49.897 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 13c11252-bb9a-403d-bf47-fa6cae3141cb 2024-04-25T07:28:49.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 13c11252-bb9a-403d-bf47-fa6cae3141cb' 2024-04-25T07:28:49.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:28:49.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxBipm8O/nNhAAxBDUgoGhTRPemPxx3uyOjg== 2024-04-25T07:28:49.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxBipm8O/nNhAAxBDUgoGhTRPemPxx3uyOjg=="}' 2024-04-25T07:28:49.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13c11252-bb9a-403d-bf47-fa6cae3141cb -i td/ec-error-rollforward/2/new.json 2024-04-25T07:28:50.471 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:28:50.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/2/new.json 2024-04-25T07:28:50.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAxBipm8O/nNhAAxBDUgoGhTRPemPxx3uyOjg== --osd-uuid 13c11252-bb9a-403d-bf47-fa6cae3141cb 2024-04-25T07:28:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:50.522+0000 7f4635c947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:50.550+0000 7f4635c947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:50.552 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:50.554+0000 7f4635c947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:50.554+0000 7f4635c947c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:28:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:50.554+0000 7f4635c947c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:28:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:50.554+0000 7f4635c947c0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2024-04-25T07:28:50.589 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:50.590+0000 7f4635c947c0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2024-04-25T07:28:57.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2024-04-25T07:28:57.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:28:57.562 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:28:57.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:28:57.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:28:58.109 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:28:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:28:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:28:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:28:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:28:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:28:58.139 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:58.138+0000 7f7945c307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:58.141 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:58.142+0000 7f7945c307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:58.143 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:58.142+0000 7f7945c307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:28:58.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:28:58.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:28:58.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:28:58.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:28:58.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:28:58.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:58.509 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:28:58.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:28:58.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:28:58.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:28:58.710 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:28:58.710+0000 7f7945c307c0 -1 Falling back to public interface 2024-04-25T07:28:58.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:28:59.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:28:59.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:28:59.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:29:00.000 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:29:00.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:00.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:00.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:01.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:01.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:01.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:29:01.395 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:29:01.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:01.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:01.447 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:01.446+0000 7f7945c307c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:29:01.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:02.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:02.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:02.857 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:29:02.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:29:02.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:02.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:03.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:04.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:04.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:04.399 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:29:04.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:29:04.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:04.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:04.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:05.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:05.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:05.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:29:05.940 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:29:05.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:05.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:06.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:07.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:07.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:07.513 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:29:07.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:29:07.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:07.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:07.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:08.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:08.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:08.982 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:29:08.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:29:08.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:08.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:09.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:10.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:10.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:10.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:29:10.542 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:29:10.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:10.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:11.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:12.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:12.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:12.016 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:29:12.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:29:12.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:12.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:12.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:13.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:13.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:13.558 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:29:13.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:29:13.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:13.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:13.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:14.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:14.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:14.993 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:29:14.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:29:14.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:14.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:16.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:17.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:17.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:17.094 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:29:17.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:29:17.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:17.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:17.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:18.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:18.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:18.855 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:29:18.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:29:18.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:18.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:19.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:20.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:20.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:20.556 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:29:20.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:29:20.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:20.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:22.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:23.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:23.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:23.125 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:29:23.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:29:23.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:23.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:23.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:24.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:24.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:24.802 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:29:24.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:29:24.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:24.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:25.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:26.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:26.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:26.354 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:29:26.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:29:26.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:26.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:27.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:28.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:28.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:28.003 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:29:28.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:29:28.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:28.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:29:28.408 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2497040542,v1:127.0.0.1:6819/2497040542] [v2:127.0.0.1:6820/2497040542,v1:127.0.0.1:6821/2497040542] exists,up 13c11252-bb9a-403d-bf47-fa6cae3141cb 2024-04-25T07:29:28.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:29:28.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:29:28.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:29:28.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2024-04-25T07:29:28.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ec-error-rollforward 2024-04-25T07:29:28.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:29:28.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T07:29:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:29:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ec-error-rollforward/3 2024-04-25T07:29:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:29:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:29:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:29:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:29:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-04-25T07:29:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-04-25T07:29:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:29:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:29:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:29:28.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:29:28.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:29:28.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:29:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:29:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:29:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:29:28.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:29:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:29:28.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:29:28.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:29:28.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ec-error-rollforward/3 2024-04-25T07:29:28.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:29:28.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f2c7330-df09-427f-b55f-4780f818dddb 2024-04-25T07:29:28.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9f2c7330-df09-427f-b55f-4780f818dddb' 2024-04-25T07:29:28.418 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 9f2c7330-df09-427f-b55f-4780f818dddb 2024-04-25T07:29:28.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:29:28.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBYBipmSxkUGhAApM9J22+6N+tzAG6i2m0FuA== 2024-04-25T07:29:28.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBYBipmSxkUGhAApM9J22+6N+tzAG6i2m0FuA=="}' 2024-04-25T07:29:28.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f2c7330-df09-427f-b55f-4780f818dddb -i td/ec-error-rollforward/3/new.json 2024-04-25T07:29:28.947 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:29:28.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ec-error-rollforward/3/new.json 2024-04-25T07:29:28.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBYBipmSxkUGhAApM9J22+6N+tzAG6i2m0FuA== --osd-uuid 9f2c7330-df09-427f-b55f-4780f818dddb 2024-04-25T07:29:28.985 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:28.989+0000 7f78e20537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:28.988 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:28.989+0000 7f78e20537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:28.989 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:28.993+0000 7f78e20537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:28.989 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:28.993+0000 7f78e20537c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:29:28.989 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:28.993+0000 7f78e20537c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:29:28.989 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:28.993+0000 7f78e20537c0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2024-04-25T07:29:29.056 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:29.057+0000 7f78e20537c0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2024-04-25T07:29:35.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2024-04-25T07:29:35.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:29:35.735 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T07:29:35.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T07:29:35.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:29:36.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T07:29:36.299 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T07:29:36.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:29:36.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:29:36.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:29:36.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:29:36.328 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:36.329+0000 7f345b1627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:36.364 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:36.365+0000 7f345b1627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:36.365 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:36.369+0000 7f345b1627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:29:36.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T07:29:36.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:29:36.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:29:36.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:29:36.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:29:36.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:36.776 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:29:36.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:29:36.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:36.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:36.942 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:36.945+0000 7f345b1627c0 -1 Falling back to public interface 2024-04-25T07:29:37.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:38.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:38.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:38.272 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:29:38.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:29:38.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:38.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:38.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:39.636 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:29:39.637+0000 7f345b1627c0 -1 osd.3 0 log_to_monitors true 2024-04-25T07:29:39.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:39.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:39.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:29:39.680 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:29:39.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:39.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:40.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:41.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:41.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:41.324 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:29:41.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:29:41.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:41.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:41.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:42.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:42.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:42.790 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:29:42.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:29:42.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:42.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:43.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:44.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:44.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:44.231 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:29:44.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:29:44.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:44.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:44.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:45.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:45.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:45.770 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:29:45.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:29:45.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:45.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:46.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:47.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:47.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:47.184 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:29:47.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:29:47.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:47.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:47.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:48.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:48.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:48.606 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:29:48.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:29:48.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:48.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:49.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:50.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:50.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:50.087 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:29:50.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:29:50.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:50.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:50.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:51.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:51.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:51.670 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:29:51.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:29:51.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:51.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:52.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:53.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:53.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:53.118 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:29:53.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:29:53.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:53.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:54.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:55.230 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:29:55.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:29:55.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:55.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:56.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:57.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:57.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:57.527 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:29:57.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:29:57.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:57.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:29:58.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:29:59.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:29:59.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:29:59.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:29:59.054 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:29:59.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:29:59.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:30:00.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:01.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:01.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:01.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:30:01.462 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:30:01.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:01.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:30:02.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:03.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:03.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:03.362 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:30:03.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:30:03.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:03.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:30:04.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:05.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:05.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:05.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:30:05.336 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:30:05.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:05.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:30:05.775 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/191137122,v1:127.0.0.1:6827/191137122] [v2:127.0.0.1:6828/191137122,v1:127.0.0.1:6829/191137122] exists,up 9f2c7330-df09-427f-b55f-4780f818dddb 2024-04-25T07:30:05.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:30:05.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:30:05.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:30:05.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-04-25T07:30:06.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2024-04-25T07:30:07.116 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'ec' already exists 2024-04-25T07:30:07.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2024-04-25T07:30:09.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2024-04-25T07:30:09.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 87748 2024-04-25T07:30:09.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=90486 2024-04-25T07:30:09.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2024-04-25T07:30:09.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2024-04-25T07:30:10.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 90507' 2024-04-25T07:30:10.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2024-04-25T07:30:10.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 90508' 2024-04-25T07:30:10.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2024-04-25T07:30:10.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 90509' 2024-04-25T07:30:10.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2024-04-25T07:30:10.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2024-04-25T07:30:11.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2024-04-25T07:30:11.205 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:30:11.205 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:30:11.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:30:11.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:30:11.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:30:11.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:30:11.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 90486 90507 90508 90509 2024-04-25T07:30:11.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2024-04-25T07:30:11.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/0 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:30:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:30:11.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:30:11.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:30:11.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:30:11.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:30:11.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:30:11.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:30:11.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:30:11.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/0 2024-04-25T07:30:11.323 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:30:11.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:30:11.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:30:11.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/0/whoami 2024-04-25T07:30:11.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:30:11.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:30:11.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:30:11.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:30:11.348 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:11.352+0000 7fe80f2aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:11.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:11.352+0000 7fe80f2aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:11.352 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:11.356+0000 7fe80f2aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:11.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:30:11.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:30:11.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:30:11.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:30:11.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:30:11.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:11.745 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:30:11.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:30:11.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:11.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:11.910 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:11.912+0000 7fe80f2aa7c0 -1 Falling back to public interface 2024-04-25T07:30:12.159 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1546806672,v1:127.0.0.1:6803/1546806672] [v2:127.0.0.1:6804/1546806672,v1:127.0.0.1:6805/1546806672] exists,up e4f34214-daa6-46aa-b6a5-7ae719353e68 2024-04-25T07:30:12.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:30:12.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:30:12.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:30:12.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2024-04-25T07:30:12.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:30:12.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:30:12.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:30:12.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:30:12.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/1 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:30:12.164 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:30:12.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/1 2024-04-25T07:30:12.166 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:30:12.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:30:12.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:30:12.167 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/1/whoami 2024-04-25T07:30:12.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:30:12.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:30:12.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:30:12.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:30:12.197 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:12.200+0000 7f8430a187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:12.211 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:12.212+0000 7f8430a187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:12.212 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:12.216+0000 7f8430a187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:12.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:30:12.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:30:12.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:30:12.598 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:30:12.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:30:12.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:30:12.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:12.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:30:12.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:12.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:30:12.790 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:12.792+0000 7f8430a187c0 -1 Falling back to public interface 2024-04-25T07:30:13.012 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3713568849,v1:127.0.0.1:6811/3713568849] [v2:127.0.0.1:6812/3713568849,v1:127.0.0.1:6813/3713568849] exists,up a3e29183-4e37-45a2-a26f-89aadbc8023b 2024-04-25T07:30:13.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:30:13.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:30:13.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:30:13.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2024-04-25T07:30:13.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/2 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:30:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:30:13.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:30:13.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:30:13.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:30:13.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:30:13.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:30:13.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:30:13.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:30:13.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/2 2024-04-25T07:30:13.019 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:30:13.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:30:13.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:30:13.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/2/whoami 2024-04-25T07:30:13.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:30:13.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:30:13.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:30:13.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:30:13.045 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:13.048+0000 7f37a22be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:13.047 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:13.048+0000 7f37a22be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:13.048 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:13.052+0000 7f37a22be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:13.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:30:13.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:30:13.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:30:13.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:30:13.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:30:13.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:13.475 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:30:13.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:30:13.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:13.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:30:13.598 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:13.600+0000 7f37a22be7c0 -1 Falling back to public interface 2024-04-25T07:30:13.886 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2497040542,v1:127.0.0.1:6819/2497040542] [v2:127.0.0.1:6820/2497040542,v1:127.0.0.1:6821/2497040542] exists,up 13c11252-bb9a-403d-bf47-fa6cae3141cb 2024-04-25T07:30:13.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:30:13.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:30:13.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:30:13.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ec-error-rollforward 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ec-error-rollforward/3 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 ' 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2024-04-25T07:30:13.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:30:13.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ec-error-rollforward/3 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:30:13.893 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T07:30:13.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=63bdc2d0-d9e7-4199-b472-939d46cdd6cb --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.83166/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:30:13.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ec-error-rollforward/3/whoami 2024-04-25T07:30:13.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:30:13.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:30:13.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:30:13.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:30:13.922 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:13.924+0000 7fef6558a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:13.925 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:13.928+0000 7fef6558a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:13.926 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:13.928+0000 7fef6558a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:14.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:30:14.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:30:14.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:30:14.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:30:14.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:30:14.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:14.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:30:14.312 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:30:14.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:14.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:30:14.458 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:14.460+0000 7fef6558a7c0 -1 Falling back to public interface 2024-04-25T07:30:14.871 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/191137122,v1:127.0.0.1:6827/191137122] [v2:127.0.0.1:6828/191137122,v1:127.0.0.1:6829/191137122] exists,up 9f2c7330-df09-427f-b55f-4780f818dddb 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:30:14.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:30:14.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:30:14.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:30:14.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:30:14.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:30:14.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:30:14.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:30:14.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:30:15.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:30:15.539 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:30:15.539 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T07:30:15.539 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T07:30:15.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:30:15.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:30:15.539 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:30:15.732 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:15.736+0000 7fe80f2aa7c0 -1 osd.0 26 log_to_monitors true 2024-04-25T07:30:16.730 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:16.732+0000 7f8430a187c0 -1 osd.1 26 log_to_monitors true 2024-04-25T07:30:17.848 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:17.848+0000 7f37a22be7c0 -1 osd.2 25 log_to_monitors true 2024-04-25T07:30:18.578 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:18.580+0000 7fef6558a7c0 -1 osd.3 26 log_to_monitors true 2024-04-25T07:30:20.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051587 2024-04-25T07:30:20.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051587 2024-04-25T07:30:20.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587' 2024-04-25T07:30:20.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:30:20.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:30:21.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953475 2024-04-25T07:30:21.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953475 2024-04-25T07:30:21.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475' 2024-04-25T07:30:21.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:30:21.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:30:21.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=146028888067 2024-04-25T07:30:21.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 146028888067 2024-04-25T07:30:21.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-146028888067' 2024-04-25T07:30:21.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:30:21.215 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:30:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=158913789955 2024-04-25T07:30:21.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 158913789955 2024-04-25T07:30:21.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051587 1-137438953475 2-146028888067 3-158913789955' 2024-04-25T07:30:21.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:30:21.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051587 2024-04-25T07:30:21.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:30:21.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:30:21.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051587 2024-04-25T07:30:21.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:30:21.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051587 2024-04-25T07:30:21.530 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 124554051587 2024-04-25T07:30:21.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051587' 2024-04-25T07:30:21.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:30:21.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 124554051587 2024-04-25T07:30:21.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:30:22.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:30:22.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:30:23.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051588 -lt 124554051587 2024-04-25T07:30:23.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:30:23.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-137438953475 2024-04-25T07:30:23.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:30:23.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:30:23.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-137438953475 2024-04-25T07:30:23.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:30:23.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953475 2024-04-25T07:30:23.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 137438953475' 2024-04-25T07:30:23.353 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 137438953475 2024-04-25T07:30:23.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:30:23.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953475 -lt 137438953475 2024-04-25T07:30:23.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:30:23.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-146028888067 2024-04-25T07:30:23.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:30:23.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:30:23.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-146028888067 2024-04-25T07:30:23.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:30:23.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=146028888067 2024-04-25T07:30:23.768 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 146028888067 2024-04-25T07:30:23.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2024-04-25T07:30:23.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:30:24.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 146028888067 -lt 146028888067 2024-04-25T07:30:24.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:30:24.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-158913789955 2024-04-25T07:30:24.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:30:24.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:30:24.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-158913789955 2024-04-25T07:30:24.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:30:24.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=158913789955 2024-04-25T07:30:24.202 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 158913789955 2024-04-25T07:30:24.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 158913789955' 2024-04-25T07:30:24.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:30:24.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 158913789955 -lt 158913789955 2024-04-25T07:30:24.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:30:24.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:30:24.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:30:25.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T07:30:25.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:30:25.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:30:25.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:30:25.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:30:25.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:30:25.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:30:25.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:30:25.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T07:30:25.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:30:25.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:30:25.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:30:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T07:30:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:30:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:30:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2024-04-25T07:30:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:30:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:30:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:30:26.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:30:26.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:30:26.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:30:26.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:30:26.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:30:26.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:30:26.287 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:30:26.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:30:26.287 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:30:26.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:30:26.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:30:26.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:30:26.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:30:26.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:30:26.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:30:26.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:30:26.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:30:26.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:30:26.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:30:26.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:30:26.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:26.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:30:26.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83166 2024-04-25T07:30:26.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:30:26.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:30:26.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:30:26.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/ec-error-rollforward 0 2024-04-25T07:30:26.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2024-04-25T07:30:26.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:30:26.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2024-04-25T07:30:26.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:30:26.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:30:26.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:30:26.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:30:26.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:30:26.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:30:26.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:30:26.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:30:26.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:30:26.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:30:26.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:30:26.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:30:26.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:30:26.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:30:26.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:30:26.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:30:26.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:30:26.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:30:26.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2024-04-25T07:30:26.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:30:26.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:26.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.83166 2024-04-25T07:30:26.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.83166 2024-04-25T07:30:26.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:30:26.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:30:26.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:30:26.320 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:30:26.320 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:30:26.369 INFO:tasks.workunit:Running workunit osd/osd-bench.sh... 2024-04-25T07:30:26.369 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-bench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh 2024-04-25T07:30:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:30:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:30:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:30:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:30:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:30:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:30:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:30:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:30:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:30:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:30:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-bench 2024-04-25T07:30:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:22: run: local dir=td/osd-bench 2024-04-25T07:30:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:23: run: shift 2024-04-25T07:30:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: export CEPH_MON=127.0.0.1:7106 2024-04-25T07:30:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:25: run: CEPH_MON=127.0.0.1:7106 2024-04-25T07:30:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:26: run: export CEPH_ARGS 2024-04-25T07:30:26.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: uuidgen 2024-04-25T07:30:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:27: run: CEPH_ARGS+='--fsid=060c87d2-c13e-4e9d-9df0-ca0f23f5b02c --auth-supported=none ' 2024-04-25T07:30:26.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7106 ' 2024-04-25T07:30:26.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:29: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-04-25T07:30:26.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: set 2024-04-25T07:30:26.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:30:26.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:31: run: local funcs=TEST_bench 2024-04-25T07:30:26.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:32: run: for func in $funcs 2024-04-25T07:30:26.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:33: run: setup td/osd-bench 2024-04-25T07:30:26.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bench 2024-04-25T07:30:26.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bench 2024-04-25T07:30:26.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:30:26.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:30:26.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:30:26.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:30:26.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:30:26.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:30:26.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:30:26.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:30:26.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:30:26.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:30:26.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:30:26.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:30:26.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:30:26.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:30:26.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:30:26.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:30:26.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:30:26.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:30:26.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:30:26.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:30:26.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:30:26.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:30:26.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:30:26.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:26.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93012 2024-04-25T07:30:26.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93012 2024-04-25T07:30:26.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:30:26.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:30:26.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bench 2024-04-25T07:30:26.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:30:26.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:26.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93012 2024-04-25T07:30:26.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.93012 2024-04-25T07:30:26.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:30:26.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:30:26.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:30:26.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:30:26.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bench 1' TERM HUP INT 2024-04-25T07:30:26.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:34: run: TEST_bench td/osd-bench 2024-04-25T07:30:26.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:40: TEST_bench: local dir=td/osd-bench 2024-04-25T07:30:26.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:42: TEST_bench: run_mon td/osd-bench a 2024-04-25T07:30:26.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bench 2024-04-25T07:30:26.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:30:26.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:30:26.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:30:26.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bench/a 2024-04-25T07:30:26.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bench/a --run-dir=td/osd-bench 2024-04-25T07:30:26.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:30:26.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:30:26.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:30:26.948 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:30:26.948 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:26.948 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93012 2024-04-25T07:30:26.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93012/$cluster-$name.asok' 2024-04-25T07:30:26.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bench/a '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.93012/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bench/log --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:30:27.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:30:27.145 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:30:27.145 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:30:27.145 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:30:27.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:30:27.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:30:27.147 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:30:27.147 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:30:27.147 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:30:27.147 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:30:27.147 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:27.147 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93012 2024-04-25T07:30:27.147 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.93012/ceph-mon.a.asok 2024-04-25T07:30:27.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:30:27.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.93012/ceph-mon.a.asok config get fsid 2024-04-25T07:30:27.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:30:27.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:30:27.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:30:27.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:30:27.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:30:27.265 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:30:27.265 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:30:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:30:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:30:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93012 2024-04-25T07:30:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.93012/ceph-mon.a.asok 2024-04-25T07:30:27.266 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:30:27.266 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.93012/ceph-mon.a.asok config get mon_host 2024-04-25T07:30:27.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:43: TEST_bench: run_mgr td/osd-bench x 2024-04-25T07:30:27.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bench 2024-04-25T07:30:27.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:30:27.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:30:27.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:30:27.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bench/x 2024-04-25T07:30:27.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:30:28.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:30:28.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:30:28.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:30:28.277 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:30:28.277 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:28.277 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93012 2024-04-25T07:30:28.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93012/$cluster-$name.asok' 2024-04-25T07:30:28.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:30:28.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bench/x '--log-file=td/osd-bench/$name.log' '--admin-socket=/tmp/ceph-asok.93012/$cluster-$name.asok' --run-dir=td/osd-bench '--pid-file=td/osd-bench/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:30:28.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:44: TEST_bench: run_osd td/osd-bench 0 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bench 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bench/0 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=060c87d2-c13e-4e9d-9df0-ca0f23f5b02c --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 ' 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bench/0' 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bench/0/journal' 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bench' 2024-04-25T07:30:28.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93012 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.93012/$cluster-$name.asok' 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.93012/$cluster-$name.asok' 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bench/$name.log' 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bench/$name.pid' 2024-04-25T07:30:28.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:30:28.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:30:28.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:30:28.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:30:28.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:30:28.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:30:28.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bench/0 2024-04-25T07:30:28.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:30:28.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1578f019-893a-4cfe-bcd8-aba07849125c 2024-04-25T07:30:28.314 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 1578f019-893a-4cfe-bcd8-aba07849125c 2024-04-25T07:30:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1578f019-893a-4cfe-bcd8-aba07849125c' 2024-04-25T07:30:28.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:30:28.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCUBipm6gLZExAAHsYIBVV62OqU9RT7/e/kSw== 2024-04-25T07:30:28.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCUBipm6gLZExAAHsYIBVV62OqU9RT7/e/kSw=="}' 2024-04-25T07:30:28.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1578f019-893a-4cfe-bcd8-aba07849125c -i td/osd-bench/0/new.json 2024-04-25T07:30:28.641 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:30:28.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bench/0/new.json 2024-04-25T07:30:28.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=060c87d2-c13e-4e9d-9df0-ca0f23f5b02c --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.93012/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUBipm6gLZExAAHsYIBVV62OqU9RT7/e/kSw== --osd-uuid 1578f019-893a-4cfe-bcd8-aba07849125c 2024-04-25T07:30:28.684 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:28.684+0000 7fb1d27907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:28.711 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:28.712+0000 7fb1d27907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:28.712 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:28.716+0000 7fb1d27907c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:28.713 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:28.716+0000 7fb1d27907c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:30:28.713 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:28.716+0000 7fb1d27907c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:30:28.713 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:28.716+0000 7fb1d27907c0 -1 bluestore(td/osd-bench/0/block) _read_bdev_label failed to open td/osd-bench/0/block: (2) No such file or directory 2024-04-25T07:30:28.749 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:28.752+0000 7fb1d27907c0 -1 bluestore(td/osd-bench/0) _read_fsid unparsable uuid 2024-04-25T07:30:35.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bench/0/keyring 2024-04-25T07:30:35.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:30:35.051 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:30:35.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:30:35.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bench/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:30:35.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:30:35.593 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:30:35.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=060c87d2-c13e-4e9d-9df0-ca0f23f5b02c --auth-supported=none --mon-host=127.0.0.1:7106 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bench/0 --osd-journal=td/osd-bench/0/journal --chdir= --run-dir=td/osd-bench '--admin-socket=/tmp/ceph-asok.93012/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bench/$name.log' '--pid-file=td/osd-bench/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:30:35.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:30:35.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:30:35.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:30:35.622 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:35.624+0000 7f7a775e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:35.658 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:35.660+0000 7f7a775e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:35.660 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:35.660+0000 7f7a775e07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:30:36.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:30:36.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:30:36.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:30:36.050 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:30:36.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:30:36.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:30:36.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:36.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:30:36.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:36.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:36.242 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:36.244+0000 7f7a775e07c0 -1 Falling back to public interface 2024-04-25T07:30:36.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:37.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:37.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:37.452 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:30:37.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:30:37.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:37.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:37.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:38.799 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:30:38.800+0000 7f7a775e07c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:30:38.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:38.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:38.859 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:30:38.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:30:38.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:38.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:39.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:40.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:40.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:40.452 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:30:40.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:30:40.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:40.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:40.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:41.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:41.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:41.939 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:30:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:30:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:41.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:42.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:43.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:43.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:43.365 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:30:43.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:30:43.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:43.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:43.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:44.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:44.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:44.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:30:44.937 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:30:44.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:44.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:45.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:46.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:46.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:46.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:30:46.405 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:30:46.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:46.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:46.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:47.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:47.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:47.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:30:47.910 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:30:47.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:47.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:48.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:49.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:49.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:49.381 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:30:49.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:30:49.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:49.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:49.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:50.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:50.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:50.944 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:30:50.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:30:50.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:50.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:51.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:52.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:52.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:52.368 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:30:52.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:30:52.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:52.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:53.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:54.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:54.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:54.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:30:54.870 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:30:54.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:54.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:56.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:57.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:57.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:57.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:30:57.123 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:30:57.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:57.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:30:58.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:30:59.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:30:59.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:30:59.233 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:30:59.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:30:59.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:30:59.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:00.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:01.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:01.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:01.055 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:31:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:31:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:01.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:02.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:02.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:02.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:31:02.909 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:31:02.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:02.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:03.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:31:04.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:31:04.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:31:04.414 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:31:04.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:31:04.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:31:04.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:31:04.851 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/103033165,v1:127.0.0.1:6803/103033165] [v2:127.0.0.1:6804/103033165,v1:127.0.0.1:6805/103033165] exists,up 1578f019-893a-4cfe-bcd8-aba07849125c 2024-04-25T07:31:04.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:31:04.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:31:04.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:31:04.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: CEPH_ARGS= 2024-04-25T07:31:04.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: ceph-conf --show-config-value osd_bench_small_size_max_iops 2024-04-25T07:31:04.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:47: TEST_bench: local osd_bench_small_size_max_iops=100 2024-04-25T07:31:04.869 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: CEPH_ARGS= 2024-04-25T07:31:04.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: ceph-conf --show-config-value osd_bench_large_size_max_throughput 2024-04-25T07:31:04.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:49: TEST_bench: local osd_bench_large_size_max_throughput=104857600 2024-04-25T07:31:04.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: CEPH_ARGS= 2024-04-25T07:31:04.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: ceph-conf --show-config-value osd_bench_max_block_size 2024-04-25T07:31:04.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:51: TEST_bench: local osd_bench_max_block_size=67108864 2024-04-25T07:31:04.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: CEPH_ARGS= 2024-04-25T07:31:04.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: ceph-conf --show-config-value osd_bench_duration 2024-04-25T07:31:04.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:53: TEST_bench: local osd_bench_duration=30 2024-04-25T07:31:04.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:58: TEST_bench: expect_failure td/osd-bench osd_bench_max_block_size ceph tell osd.0 bench 1024 67108865 2024-04-25T07:31:04.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:31:04.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:31:04.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=osd_bench_max_block_size 2024-04-25T07:31:04.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:31:04.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:31:04.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 1024 67108865 2024-04-25T07:31:05.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:31:05.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:31:05.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet osd_bench_max_block_size td/osd-bench/out 2024-04-25T07:31:05.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:31:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:64: TEST_bench: local bsize=1024 2024-04-25T07:31:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:65: TEST_bench: local max_count=3072000 2024-04-25T07:31:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:66: TEST_bench: expect_failure td/osd-bench bench_small_size_max_iops ceph tell osd.0 bench 3072001 1024 2024-04-25T07:31:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:31:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:31:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=bench_small_size_max_iops 2024-04-25T07:31:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:31:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:31:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 3072001 1024 2024-04-25T07:31:05.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:31:05.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:31:05.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet bench_small_size_max_iops td/osd-bench/out 2024-04-25T07:31:05.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:31:05.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:72: TEST_bench: local bsize=1048577 2024-04-25T07:31:05.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:73: TEST_bench: local max_count=3145728000 2024-04-25T07:31:05.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:74: TEST_bench: expect_failure td/osd-bench osd_bench_large_size_max_throughput ceph tell osd.0 bench 3145728001 1048577 2024-04-25T07:31:05.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: expect_failure: local dir=td/osd-bench 2024-04-25T07:31:05.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1979: expect_failure: shift 2024-04-25T07:31:05.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: expect_failure: local expected=osd_bench_large_size_max_throughput 2024-04-25T07:31:05.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: expect_failure: shift 2024-04-25T07:31:05.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: expect_failure: local success 2024-04-25T07:31:05.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: expect_failure: ceph tell osd.0 bench 3145728001 1048577 2024-04-25T07:31:05.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: expect_failure: success=false 2024-04-25T07:31:05.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: false 2024-04-25T07:31:05.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: expect_failure: grep --quiet osd_bench_large_size_max_throughput td/osd-bench/out 2024-04-25T07:31:05.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: expect_failure: return 0 2024-04-25T07:31:05.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:80: TEST_bench: ceph tell osd.0 bench 2024-04-25T07:32:00.576 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:32:00.576 INFO:tasks.workunit.client.0.smithi134.stdout: "bytes_written": 1073741824, 2024-04-25T07:32:00.576 INFO:tasks.workunit.client.0.smithi134.stdout: "blocksize": 4194304, 2024-04-25T07:32:00.577 INFO:tasks.workunit.client.0.smithi134.stdout: "elapsed_sec": 26.181034428, 2024-04-25T07:32:00.577 INFO:tasks.workunit.client.0.smithi134.stdout: "bytes_per_sec": 41012200.146364667, 2024-04-25T07:32:00.577 INFO:tasks.workunit.client.0.smithi134.stdout: "iops": 9.7780704847251574 2024-04-25T07:32:00.577 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:32:00.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:84: TEST_bench: ceph tell osd.0 bench 10 14456 4444 3 2024-04-25T07:32:01.118 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:32:01.118 INFO:tasks.workunit.client.0.smithi134.stdout: "bytes_written": 10, 2024-04-25T07:32:01.118 INFO:tasks.workunit.client.0.smithi134.stdout: "blocksize": 14456, 2024-04-25T07:32:01.118 INFO:tasks.workunit.client.0.smithi134.stdout: "elapsed_sec": 0.041777319, 2024-04-25T07:32:01.118 INFO:tasks.workunit.client.0.smithi134.stdout: "bytes_per_sec": 239.36433067904622, 2024-04-25T07:32:01.119 INFO:tasks.workunit.client.0.smithi134.stdout: "iops": 0.016558130235130478 2024-04-25T07:32:01.119 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:32:01.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:90: TEST_bench: ceph tell osd.0 bench 1 14456 2024-04-25T07:32:01.511 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T07:32:01.511 INFO:tasks.workunit.client.0.smithi134.stdout: "bytes_written": 1, 2024-04-25T07:32:01.511 INFO:tasks.workunit.client.0.smithi134.stdout: "blocksize": 14456, 2024-04-25T07:32:01.511 INFO:tasks.workunit.client.0.smithi134.stdout: "elapsed_sec": 0.074781644999999994, 2024-04-25T07:32:01.511 INFO:tasks.workunit.client.0.smithi134.stdout: "bytes_per_sec": 13.37226534666361, 2024-04-25T07:32:01.511 INFO:tasks.workunit.client.0.smithi134.stdout: "iops": 0.0009250321905550367 2024-04-25T07:32:01.511 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T07:32:01.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bench.sh:35: run: teardown td/osd-bench 2024-04-25T07:32:01.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:32:01.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:32:01.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:32:01.539 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:32:01.539 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:32:01.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:32:01.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:32:01.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:32:01.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:32:01.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:32:01.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:32:01.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:32:01.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:32:01.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:32:01.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:32:01.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:32:01.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:32:01.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:32:01.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:32:01.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:32:01.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:32:01.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:32:01.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:32:01.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:01.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93012 2024-04-25T07:32:01.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93012 2024-04-25T07:32:01.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-bench 0 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bench 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bench KILL 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:32:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:32:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:32:01.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:32:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:32:01.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:32:01.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:32:01.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:32:01.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:32:01.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:32:01.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:32:01.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:32:01.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:32:01.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:32:01.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:32:01.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bench 2024-04-25T07:32:01.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:32:01.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:01.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.93012 2024-04-25T07:32:01.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.93012 2024-04-25T07:32:01.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:32:01.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:32:01.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:32:01.694 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:32:01.694 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:32:01.745 INFO:tasks.workunit:Running workunit osd/osd-bluefs-volume-ops.sh... 2024-04-25T07:32:01.746 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-bluefs-volume-ops.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh 2024-04-25T07:32:01.800 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:32:01.807 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:32:01.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:32:01.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:32:01.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:32:01.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:32:01.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:32:01.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:32:01.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:32:01.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:32:01.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-bluefs-volume-ops 2024-04-25T07:32:01.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:8: run: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:01.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:9: run: shift 2024-04-25T07:32:01.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: set 2024-04-25T07:32:01.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:32:01.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:11: run: local 'funcs=TEST_bluestore 2024-04-25T07:32:01.810 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_bluestore2 2024-04-25T07:32:01.810 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_bluestore_expand' 2024-04-25T07:32:01.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:32:01.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:32:01.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:32:01.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:32:01.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:32:01.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:32:01.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:32:01.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:32:01.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:32:01.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:32:01.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:32:01.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:32:01.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:32:01.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:32:01.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:32:01.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:32:01.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:32:01.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:01.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:32:01.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95185 2024-04-25T07:32:01.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:32:01.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:32:01.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:32:01.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:32:01.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:01.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:32:01.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.95185 2024-04-25T07:32:01.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:32:01.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:32:01.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:32:01.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:32:01.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:32:01.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore td/osd-bluefs-volume-ops 2024-04-25T07:32:01.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:20: TEST_bluestore: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:01.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: ulimit -n 2024-04-25T07:32:01.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:22: TEST_bluestore: local flimit=4096 2024-04-25T07:32:01.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:23: TEST_bluestore: '[' 4096 -lt 1536 ']' 2024-04-25T07:32:01.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:32:01.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:26: TEST_bluestore: CEPH_MON=127.0.0.1:7146 2024-04-25T07:32:01.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:27: TEST_bluestore: export CEPH_ARGS 2024-04-25T07:32:01.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: uuidgen 2024-04-25T07:32:01.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:28: TEST_bluestore: CEPH_ARGS+='--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none ' 2024-04-25T07:32:01.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:29: TEST_bluestore: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:32:01.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:30: TEST_bluestore: CEPH_ARGS+='--bluestore_block_size=2147483648 ' 2024-04-25T07:32:01.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:31: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:32:01.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:32: TEST_bluestore: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:32:01.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:33: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_size=536870912 ' 2024-04-25T07:32:01.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:34: TEST_bluestore: CEPH_ARGS+='--bluestore_block_wal_create=true ' 2024-04-25T07:32:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:35: TEST_bluestore: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:32:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:37: TEST_bluestore: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:32:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:32:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:32:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:32:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:32:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:02.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:32:02.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:32:02.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:32:02.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:32:02.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:02.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:32:02.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:32:02.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:32:02.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:32:02.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:32:02.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:32:02.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:32:02.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:32:02.511 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:32:02.511 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:32:02.511 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:32:02.511 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:32:02.511 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:32:02.511 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:02.511 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:32:02.511 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95185/ceph-mon.a.asok 2024-04-25T07:32:02.511 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:32:02.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95185/ceph-mon.a.asok config get fsid 2024-04-25T07:32:02.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:32:02.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:32:02.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:32:02.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:32:02.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:32:02.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:32:02.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:32:02.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:32:02.629 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:32:02.629 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:02.629 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:32:02.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95185/ceph-mon.a.asok 2024-04-25T07:32:02.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:32:02.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95185/ceph-mon.a.asok config get mon_host 2024-04-25T07:32:02.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:38: TEST_bluestore: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:32:02.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:02.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:32:02.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:32:02.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:32:02.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:32:02.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:32:03.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:32:03.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:32:03.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:32:03.634 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:32:03.634 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:03.634 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:32:03.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:32:03.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:32:03.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:32:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:39: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:32:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:32:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:32:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:32:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:32:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:32:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:32:03.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:32:03.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:32:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:32:03.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:32:03.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2d96bda-b606-4fae-8541-2faf9c7a0ded 2024-04-25T07:32:03.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a2d96bda-b606-4fae-8541-2faf9c7a0ded' 2024-04-25T07:32:03.672 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 a2d96bda-b606-4fae-8541-2faf9c7a0ded 2024-04-25T07:32:03.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:32:03.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDzBipmQLNEKRAAwwC9HSvTIhsdtQ3vYEAPxQ== 2024-04-25T07:32:03.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDzBipmQLNEKRAAwwC9HSvTIhsdtQ3vYEAPxQ=="}' 2024-04-25T07:32:03.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2d96bda-b606-4fae-8541-2faf9c7a0ded -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:32:03.969 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:32:03.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:32:03.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDzBipmQLNEKRAAwwC9HSvTIhsdtQ3vYEAPxQ== --osd-uuid a2d96bda-b606-4fae-8541-2faf9c7a0ded 2024-04-25T07:32:04.011 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:04.010+0000 7f384fe0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:04.038 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:04.038+0000 7f384fe0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:04.039 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:04.038+0000 7f384fe0e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:04.039 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:04.038+0000 7f384fe0e7c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:32:04.039 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:04.038+0000 7f384fe0e7c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:32:04.039 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:04.038+0000 7f384fe0e7c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:32:04.078 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:04.078+0000 7f384fe0e7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:32:14.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:32:14.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:32:14.287 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:32:14.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:32:14.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:32:14.832 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:32:14.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:32:14.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:32:14.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:32:14.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:32:14.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:32:14.862 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:14.862+0000 7f91bdeba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:14.864 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:14.866+0000 7f91bdeba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:14.865 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:14.866+0000 7f91bdeba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:15.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:32:15.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:32:15.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:32:15.224 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:32:15.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:32:15.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:32:15.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:15.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:32:15.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:15.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:15.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:15.899 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:15.898+0000 7f91bdeba7c0 -1 Falling back to public interface 2024-04-25T07:32:16.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:16.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:16.659 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:32:16.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:32:16.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:16.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:17.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:18.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:18.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:18.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:32:18.075 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:32:18.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:18.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:18.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:19.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:19.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:19.492 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:32:19.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:32:19.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:19.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:19.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:20.710 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:20.714+0000 7f91bdeba7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:32:20.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:20.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:20.904 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:32:20.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:32:20.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:20.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:21.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:22.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:22.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:22.434 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:32:22.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:32:22.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:22.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:22.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:23.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:23.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:23.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:32:23.925 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:32:23.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:23.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:24.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:25.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:25.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:25.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:32:25.383 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:32:25.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:25.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:25.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:26.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:26.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:26.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:32:26.862 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:32:26.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:26.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:27.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:28.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:28.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:28.301 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:32:28.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:32:28.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:28.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:29.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:30.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:30.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:30.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:32:30.031 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:32:30.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:30.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:30.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:31.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:31.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:31.464 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:32:31.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:32:31.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:31.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:31.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:32.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:32.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:32.943 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:32:32.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:32:32.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:32.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:33.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:34.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:34.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:34.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:32:34.365 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:32:34.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:34.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:35.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:36.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:36.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:32:36.226 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:32:36.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:36.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:37.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:38.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:38.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:38.318 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:32:38.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:32:38.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:38.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:39.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:40.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:40.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:40.165 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:32:40.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:32:40.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:40.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:40.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:41.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:41.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:41.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:32:41.997 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:32:41.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:41.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:42.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:43.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:43.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:43.785 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:32:43.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:32:43.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:43.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:44.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:45.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:45.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:45.744 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:32:45.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:32:45.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:45.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:32:46.174 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/186982862,v1:127.0.0.1:6803/186982862] [v2:127.0.0.1:6804/186982862,v1:127.0.0.1:6805/186982862] exists,up a2d96bda-b606-4fae-8541-2faf9c7a0ded 2024-04-25T07:32:46.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:32:46.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:32:46.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:32:46.175 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:32:46.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:40: TEST_bluestore: osd_pid0=96646 2024-04-25T07:32:46.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:41: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:32:46.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:32:46.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:32:46.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:32:46.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:32:46.180 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:32:46.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:32:46.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:32:46.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:32:46.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac851636-c635-4577-a7fd-b89070ff1f35 2024-04-25T07:32:46.182 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 ac851636-c635-4577-a7fd-b89070ff1f35 2024-04-25T07:32:46.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ac851636-c635-4577-a7fd-b89070ff1f35' 2024-04-25T07:32:46.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:32:46.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeBypm5pRcDBAApt2J2wwz/yDAOp5Xlx0/tw== 2024-04-25T07:32:46.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeBypm5pRcDBAApt2J2wwz/yDAOp5Xlx0/tw=="}' 2024-04-25T07:32:46.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac851636-c635-4577-a7fd-b89070ff1f35 -i td/osd-bluefs-volume-ops/1/new.json 2024-04-25T07:32:46.661 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:32:46.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/1/new.json 2024-04-25T07:32:46.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeBypm5pRcDBAApt2J2wwz/yDAOp5Xlx0/tw== --osd-uuid ac851636-c635-4577-a7fd-b89070ff1f35 2024-04-25T07:32:46.706 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:46.709+0000 7fad334be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:46.708 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:46.713+0000 7fad334be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:46.709 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:46.713+0000 7fad334be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:46.710 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:46.713+0000 7fad334be7c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:32:46.710 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:46.713+0000 7fad334be7c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:32:46.710 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:46.713+0000 7fad334be7c0 -1 bluestore(td/osd-bluefs-volume-ops/1/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/1/block: (2) No such file or directory 2024-04-25T07:32:46.786 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:46.789+0000 7fad334be7c0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_fsid unparsable uuid 2024-04-25T07:32:57.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/1/keyring 2024-04-25T07:32:57.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:32:57.532 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:32:57.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:32:57.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:32:58.083 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:32:58.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:32:58.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:32:58.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:32:58.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:32:58.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:32:58.113 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:58.117+0000 7f571045e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:58.141 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:58.145+0000 7f571045e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:58.142 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:58.145+0000 7f571045e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:32:58.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:32:58.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:32:58.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:32:58.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:32:58.489 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:32:58.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:32:58.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:58.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:32:58.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:58.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:32:58.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:32:59.191 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:32:59.193+0000 7f571045e7c0 -1 Falling back to public interface 2024-04-25T07:32:59.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:32:59.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:32:59.919 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:32:59.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:32:59.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:32:59.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:00.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:01.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:01.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:01.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:01.352 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:33:01.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:01.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:01.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:02.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:02.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:02.752 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:33:02.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:33:02.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:02.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:03.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:04.084 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:04.089+0000 7f571045e7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:33:04.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:04.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:04.162 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:33:04.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:33:04.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:04.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:04.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:05.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:05.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:05.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:33:05.807 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:33:05.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:05.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:06.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:07.243 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:33:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:33:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:07.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:08.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:08.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:08.694 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:33:08.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:33:08.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:08.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:09.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:10.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:10.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:10.179 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:33:10.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:33:10.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:10.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:10.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:11.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:11.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:11.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:33:11.665 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:33:11.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:11.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:12.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:13.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:13.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:13.317 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:33:13.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:33:13.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:13.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:13.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:14.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:14.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:14.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:33:14.756 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:33:14.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:14.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:15.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:16.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:16.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:16.387 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:33:16.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:33:16.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:16.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:16.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:17.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:17.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:17.825 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:33:17.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:33:17.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:17.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:18.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:19.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:19.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:19.737 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:33:19.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:33:19.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:19.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:20.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:21.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:21.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:21.369 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:33:21.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:33:21.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:21.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:22.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:23.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:23.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:23.205 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:33:23.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:33:23.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:23.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:24.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:25.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:25.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:25.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:33:25.043 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:33:25.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:25.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:25.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:26.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:26.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:26.878 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:33:26.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:33:26.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:26.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:27.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:28.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:28.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:28.799 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:33:28.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:33:28.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:28.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:29.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:30.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:30.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:30.248 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:33:30.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:33:30.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:30.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:33:30.705 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/459835881,v1:127.0.0.1:6811/459835881] [v2:127.0.0.1:6812/459835881,v1:127.0.0.1:6813/459835881] exists,up ac851636-c635-4577-a7fd-b89070ff1f35 2024-04-25T07:33:30.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:33:30.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:33:30.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:33:30.706 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:33:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:42: TEST_bluestore: osd_pid1=99187 2024-04-25T07:33:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:43: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:33:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:33:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:33:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:33:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:33:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:33:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:33:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:33:30.712 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:33:30.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:33:30.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:33:30.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2febbe7e-a53e-4e2d-9cb9-0f565790e7f7 2024-04-25T07:33:30.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2febbe7e-a53e-4e2d-9cb9-0f565790e7f7' 2024-04-25T07:33:30.715 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 2febbe7e-a53e-4e2d-9cb9-0f565790e7f7 2024-04-25T07:33:30.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:33:30.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKBypm3dwCLBAAg4h3Yrt5ft703kng9Uoqaw== 2024-04-25T07:33:30.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKBypm3dwCLBAAg4h3Yrt5ft703kng9Uoqaw=="}' 2024-04-25T07:33:30.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2febbe7e-a53e-4e2d-9cb9-0f565790e7f7 -i td/osd-bluefs-volume-ops/2/new.json 2024-04-25T07:33:31.196 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:33:31.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/2/new.json 2024-04-25T07:33:31.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKBypm3dwCLBAAg4h3Yrt5ft703kng9Uoqaw== --osd-uuid 2febbe7e-a53e-4e2d-9cb9-0f565790e7f7 2024-04-25T07:33:31.250 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:31.252+0000 7f68868ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:31.281 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:31.284+0000 7f68868ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:31.282 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:31.284+0000 7f68868ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:31.283 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:31.284+0000 7f68868ab7c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:33:31.283 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:31.284+0000 7f68868ab7c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:33:31.283 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:31.284+0000 7f68868ab7c0 -1 bluestore(td/osd-bluefs-volume-ops/2/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/2/block: (2) No such file or directory 2024-04-25T07:33:31.321 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:31.324+0000 7f68868ab7c0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_fsid unparsable uuid 2024-04-25T07:33:41.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/2/keyring 2024-04-25T07:33:41.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:33:41.950 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:33:41.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:33:41.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:33:42.506 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:33:42.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:33:42.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:33:42.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:33:42.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:33:42.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:33:42.530 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:42.532+0000 7fa1f08f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:42.532 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:42.536+0000 7fa1f08f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:42.533 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:42.536+0000 7fa1f08f67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:33:42.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:33:42.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:33:42.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:33:42.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:33:42.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:33:42.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:42.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:33:42.948 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:33:42.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:42.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:43.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:43.566 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:43.568+0000 7fa1f08f67c0 -1 Falling back to public interface 2024-04-25T07:33:44.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:44.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:44.391 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:33:44.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:33:44.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:44.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:44.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:45.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:45.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:45.855 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:33:45.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:33:45.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:45.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:46.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:47.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:47.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:47.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:33:47.279 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:33:47.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:47.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:47.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:48.542 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:33:48.544+0000 7fa1f08f67c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:33:48.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:48.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:48.688 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:33:48.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:33:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:49.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:50.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:50.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:50.333 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:33:50.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:33:50.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:50.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:51.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:52.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:52.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:52.070 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:33:52.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:33:52.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:52.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:52.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:53.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:53.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:53.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:33:53.633 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:33:53.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:53.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:54.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:55.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:55.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:55.084 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:33:55.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:33:55.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:55.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:55.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:56.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:56.731 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:33:56.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:56.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:33:56.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:56.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:57.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:33:58.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:33:58.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:33:58.281 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:33:58.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:33:58.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:33:58.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:33:59.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:00.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:00.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:00.034 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:34:00.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:34:00.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:00.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:34:00.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:01.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:01.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:01.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:34:01.643 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:34:01.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:01.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:34:02.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:03.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:03.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:03.153 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:34:03.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:34:03.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:03.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:34:04.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:05.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:05.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:05.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:34:05.239 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:34:05.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:05.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:34:06.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:07.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:07.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:07.196 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:34:07.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:34:07.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:07.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:34:07.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:08.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:08.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:08.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:34:08.950 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:34:08.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:08.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:34:09.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:10.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:10.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:10.855 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:34:10.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:34:10.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:10.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:34:11.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:12.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:12.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:12.652 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:34:12.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:34:12.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:12.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:34:13.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:14.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:14.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:14.484 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:34:14.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:34:14.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:14.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:34:14.969 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2778975195,v1:127.0.0.1:6819/2778975195] [v2:127.0.0.1:6820/2778975195,v1:127.0.0.1:6821/2778975195] exists,up 2febbe7e-a53e-4e2d-9cb9-0f565790e7f7 2024-04-25T07:34:14.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:34:14.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:34:14.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:34:14.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:44: TEST_bluestore: osd_pid2=101753 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:45: TEST_bluestore: run_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:34:14.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:34:14.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:34:14.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:34:14.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:34:14.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:34:14.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:34:14.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:34:14.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:34:14.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:34:14.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:34:14.975 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:34:14.975 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:34:14.975 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:34:14.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:34:14.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:34:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:34:14.977 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:34:14.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b4287a97-f3f4-4057-a2f6-cd858e9a523b 2024-04-25T07:34:14.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b4287a97-f3f4-4057-a2f6-cd858e9a523b' 2024-04-25T07:34:14.977 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 b4287a97-f3f4-4057-a2f6-cd858e9a523b 2024-04-25T07:34:14.978 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:34:14.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB3Bypmqg8vABAAP8XCBWDrsbxjK0uB3Xg0AA== 2024-04-25T07:34:14.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB3Bypmqg8vABAAP8XCBWDrsbxjK0uB3Xg0AA=="}' 2024-04-25T07:34:14.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b4287a97-f3f4-4057-a2f6-cd858e9a523b -i td/osd-bluefs-volume-ops/3/new.json 2024-04-25T07:34:15.508 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:34:15.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/3/new.json 2024-04-25T07:34:15.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB3Bypmqg8vABAAP8XCBWDrsbxjK0uB3Xg0AA== --osd-uuid b4287a97-f3f4-4057-a2f6-cd858e9a523b 2024-04-25T07:34:15.559 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:15.559+0000 7f4a4ba337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:15.628 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:15.631+0000 7f4a4ba337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:15.630 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:15.631+0000 7f4a4ba337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:15.630 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:15.631+0000 7f4a4ba337c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:34:15.631 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:15.631+0000 7f4a4ba337c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:34:15.631 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:15.631+0000 7f4a4ba337c0 -1 bluestore(td/osd-bluefs-volume-ops/3/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/3/block: (2) No such file or directory 2024-04-25T07:34:15.762 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:15.763+0000 7f4a4ba337c0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_fsid unparsable uuid 2024-04-25T07:34:26.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/3/keyring 2024-04-25T07:34:26.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:34:26.504 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T07:34:26.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T07:34:26.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:34:27.049 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T07:34:27.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T07:34:27.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:34:27.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:34:27.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:34:27.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:34:27.080 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:27.083+0000 7ff446afe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:27.082 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:27.083+0000 7ff446afe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:27.084 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:27.083+0000 7ff446afe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:34:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T07:34:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:34:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:34:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:34:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:34:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:34:27.458 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:34:27.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:27.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:27.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:28.123 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:28.123+0000 7ff446afe7c0 -1 Falling back to public interface 2024-04-25T07:34:28.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:28.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:28.874 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:34:28.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:34:28.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:28.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:29.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:30.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:30.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:30.273 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:34:30.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:34:30.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:30.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:30.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:31.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:31.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:31.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:34:31.678 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:34:31.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:31.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:32.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:33.069 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:34:33.070+0000 7ff446afe7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T07:34:33.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:33.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:33.086 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:34:33.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:34:33.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:33.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:33.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:34.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:34.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:34.871 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:34:34.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:34:34.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:34.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:35.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:36.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:36.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:36.282 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:34:36.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:34:36.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:36.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:36.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:37.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:37.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:37.748 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:34:37.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:34:37.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:37.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:38.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:39.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:39.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:39.299 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:34:39.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:34:39.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:39.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:39.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:40.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:40.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:40.762 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:34:40.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:34:40.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:40.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:41.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:42.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:42.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:42.365 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:34:42.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:34:42.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:42.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:42.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:43.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:43.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:43.786 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:34:43.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:34:43.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:43.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:44.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:45.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:45.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:45.299 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:34:45.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:34:45.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:45.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:45.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:46.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:46.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:46.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:34:46.748 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:34:46.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:46.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:47.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:48.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:48.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:48.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:34:48.763 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:34:48.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:48.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:49.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:50.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:50.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:50.426 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:34:50.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:34:50.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:50.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:50.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:51.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:51.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:51.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:34:51.869 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:34:51.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:51.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:52.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:53.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:53.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:53.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:34:53.661 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:34:53.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:53.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:54.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:55.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:55.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:55.809 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:34:55.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:34:55.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:55.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:56.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:57.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:57.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:57.726 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:34:57.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:34:57.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:57.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:58.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:34:59.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:34:59.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:34:59.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:34:59.294 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:34:59.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:34:59.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:34:59.719 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/22951723,v1:127.0.0.1:6827/22951723] [v2:127.0.0.1:6828/22951723,v1:127.0.0.1:6829/22951723] exists,up b4287a97-f3f4-4057-a2f6-cd858e9a523b 2024-04-25T07:34:59.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:34:59.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:34:59.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:34:59.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:34:59.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:46: TEST_bluestore: osd_pid3=104294 2024-04-25T07:34:59.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:48: TEST_bluestore: sleep 5 2024-04-25T07:35:04.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:50: TEST_bluestore: create_pool foo 16 2024-04-25T07:35:04.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:35:05.275 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T07:35:05.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:35:06.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:53: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:35:06.566 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T07:35:06.566 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:35:06.566 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_105679 2024-04-25T07:35:06.567 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:35:06.567 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:35:07.567 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 38 22 0.0859292 0.0859375 0.321901 0.607809 2024-04-25T07:35:08.567 INFO:tasks.workunit.client.0.smithi134.stdout: 2 16 84 68 0.132799 0.179688 0.404702 0.416718 2024-04-25T07:35:09.567 INFO:tasks.workunit.client.0.smithi134.stdout: 3 16 137 121 0.157536 0.207031 0.308619 0.380089 2024-04-25T07:35:10.567 INFO:tasks.workunit.client.0.smithi134.stdout: 4 16 190 174 0.169903 0.207031 0.283526 0.35407 2024-04-25T07:35:11.567 INFO:tasks.workunit.client.0.smithi134.stdout: 5 16 247 231 0.180449 0.222656 0.293536 0.339693 2024-04-25T07:35:12.567 INFO:tasks.workunit.client.0.smithi134.stdout: 6 16 297 281 0.182922 0.195312 0.298187 0.333714 2024-04-25T07:35:13.567 INFO:tasks.workunit.client.0.smithi134.stdout: 7 16 351 335 0.18692 0.210938 0.30944 0.328825 2024-04-25T07:35:14.567 INFO:tasks.workunit.client.0.smithi134.stdout: 8 16 399 383 0.186989 0.1875 0.300725 0.328669 2024-04-25T07:35:15.568 INFO:tasks.workunit.client.0.smithi134.stdout: 9 16 448 432 0.187476 0.191406 0.300081 0.325507 2024-04-25T07:35:16.568 INFO:tasks.workunit.client.0.smithi134.stdout: 10 16 505 489 0.190992 0.222656 0.316779 0.322876 2024-04-25T07:35:17.568 INFO:tasks.workunit.client.0.smithi134.stdout: 11 16 558 542 0.192448 0.207031 0.308641 0.320994 2024-04-25T07:35:18.568 INFO:tasks.workunit.client.0.smithi134.stdout: 12 16 606 590 0.192033 0.1875 0.392526 0.320271 2024-04-25T07:35:19.568 INFO:tasks.workunit.client.0.smithi134.stdout: 13 16 654 638 0.191682 0.1875 0.326894 0.321569 2024-04-25T07:35:20.568 INFO:tasks.workunit.client.0.smithi134.stdout: 14 16 707 691 0.192776 0.207031 0.320949 0.319509 2024-04-25T07:35:21.569 INFO:tasks.workunit.client.0.smithi134.stdout: 15 16 766 750 0.195282 0.230469 0.309115 0.318633 2024-04-25T07:35:22.569 INFO:tasks.workunit.client.0.smithi134.stdout: 16 16 814 798 0.194794 0.1875 0.302195 0.317342 2024-04-25T07:35:23.569 INFO:tasks.workunit.client.0.smithi134.stdout: 17 16 863 847 0.194593 0.191406 0.292052 0.317274 2024-04-25T07:35:24.569 INFO:tasks.workunit.client.0.smithi134.stdout: 18 16 919 903 0.195933 0.21875 0.274986 0.316749 2024-04-25T07:35:25.570 INFO:tasks.workunit.client.0.smithi134.stdout: 19 16 969 953 0.195899 0.195312 0.297577 0.316046 2024-04-25T07:35:26.570 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:35:26.575434+0000 min lat: 0.21891 max lat: 0.800546 avg lat: 0.315101 2024-04-25T07:35:26.570 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:35:26.570 INFO:tasks.workunit.client.0.smithi134.stdout: 20 16 1023 1007 0.196649 0.210938 0.333769 0.315101 2024-04-25T07:35:27.570 INFO:tasks.workunit.client.0.smithi134.stdout: 21 16 1073 1057 0.196584 0.195312 0.288765 0.315091 2024-04-25T07:35:28.570 INFO:tasks.workunit.client.0.smithi134.stdout: 22 16 1122 1106 0.196348 0.191406 0.324909 0.316536 2024-04-25T07:35:29.570 INFO:tasks.workunit.client.0.smithi134.stdout: 23 16 1170 1154 0.195962 0.1875 0.285503 0.317334 2024-04-25T07:35:30.570 INFO:tasks.workunit.client.0.smithi134.stdout: 24 16 1218 1202 0.195608 0.1875 0.317087 0.317798 2024-04-25T07:35:31.570 INFO:tasks.workunit.client.0.smithi134.stdout: 25 16 1266 1250 0.195283 0.1875 0.325472 0.317319 2024-04-25T07:35:32.570 INFO:tasks.workunit.client.0.smithi134.stdout: 26 16 1325 1309 0.196635 0.230469 0.293247 0.315869 2024-04-25T07:35:33.571 INFO:tasks.workunit.client.0.smithi134.stdout: 27 16 1373 1357 0.196295 0.1875 0.302216 0.315862 2024-04-25T07:35:34.571 INFO:tasks.workunit.client.0.smithi134.stdout: 28 16 1395 1379 0.192354 0.0859375 0.327488 0.322276 2024-04-25T07:35:35.571 INFO:tasks.workunit.client.0.smithi134.stdout: 29 16 1443 1427 0.192185 0.1875 0.301948 0.322458 2024-04-25T07:35:36.571 INFO:tasks.workunit.client.0.smithi134.stdout: 30 16 1501 1485 0.19333 0.226562 0.298795 0.321738 2024-04-25T07:35:37.571 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 30.1989 2024-04-25T07:35:37.571 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 1501 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 4096 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 4096 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 0.194156 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 0.0327905 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 0.230469 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 0.0859375 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 49 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 8.39437 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 59 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 22 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 0.321325 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.0775595 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 0.888051 2024-04-25T07:35:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.21891 2024-04-25T07:35:37.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:55: TEST_bluestore: echo 'after bench' 2024-04-25T07:35:37.697 INFO:tasks.workunit.client.0.smithi134.stdout:after bench 2024-04-25T07:35:37.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 96646 2024-04-25T07:35:37.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: sleep 1 2024-04-25T07:35:38.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:58: TEST_bluestore: kill 96646 2024-04-25T07:35:38.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 58: kill: (96646) - No such process 2024-04-25T07:35:38.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:59: TEST_bluestore: ceph osd down 0 2024-04-25T07:35:39.260 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:35:39.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 99187 2024-04-25T07:35:39.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: sleep 1 2024-04-25T07:35:40.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:60: TEST_bluestore: kill 99187 2024-04-25T07:35:40.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 60: kill: (99187) - No such process 2024-04-25T07:35:40.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:61: TEST_bluestore: ceph osd down 1 2024-04-25T07:35:40.819 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already down. 2024-04-25T07:35:40.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 101753 2024-04-25T07:35:40.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: sleep 1 2024-04-25T07:35:41.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:62: TEST_bluestore: kill 101753 2024-04-25T07:35:41.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 62: kill: (101753) - No such process 2024-04-25T07:35:41.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:63: TEST_bluestore: ceph osd down 2 2024-04-25T07:35:42.270 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already down. 2024-04-25T07:35:42.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 104294 2024-04-25T07:35:42.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: sleep 1 2024-04-25T07:35:43.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:64: TEST_bluestore: kill 104294 2024-04-25T07:35:43.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 64: kill: (104294) - No such process 2024-04-25T07:35:43.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:65: TEST_bluestore: ceph osd down 3 2024-04-25T07:35:43.832 INFO:tasks.workunit.client.0.smithi134.stderr:osd.3 is already down. 2024-04-25T07:35:43.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:68: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:35:45.660 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:35:45.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:69: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:35:47.493 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:35:47.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:70: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:35:49.316 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:35:49.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:71: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:35:51.168 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:35:51.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:73: TEST_bluestore: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-04-25T07:35:51.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:74: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-04-25T07:35:51.229 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:52.120 INFO:tasks.workunit.client.0.smithi134.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:35:52.120 INFO:tasks.workunit.client.0.smithi134.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:35:52.120 INFO:tasks.workunit.client.0.smithi134.stdout:2 : device size 0x100000000 : using 0x804a0000(2.0 GiB) 2024-04-25T07:35:52.120 INFO:tasks.workunit.client.0.smithi134.stdout:Expanding DB/WAL... 2024-04-25T07:35:52.120 INFO:tasks.workunit.client.0.smithi134.stdout:0 : expanding to 0x536870912 2024-04-25T07:35:52.149 INFO:tasks.workunit.client.0.smithi134.stdout:0 : size label updated to 536870912 2024-04-25T07:35:52.149 INFO:tasks.workunit.client.0.smithi134.stdout:1 : expanding to 0x1073741824 2024-04-25T07:35:52.174 INFO:tasks.workunit.client.0.smithi134.stdout:1 : size label updated to 1073741824 2024-04-25T07:35:52.174 INFO:tasks.workunit.client.0.smithi134.stdout:2 : expanding from 0x80000000 to 0x100000000 2024-04-25T07:35:52.467 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:35:52.472+0000 7f0cb9f15ac0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-04-25T07:35:52.468 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:35:52.472+0000 7f0cb9f15ac0 -1 bluestore(td/osd-bluefs-volume-ops/0) unable to read label for td/osd-bluefs-volume-ops/0: (21) Is a directory 2024-04-25T07:35:57.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:75: TEST_bluestore: truncate td/osd-bluefs-volume-ops/1/block -s 4311744512 2024-04-25T07:35:57.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:76: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 bluefs-bdev-expand 2024-04-25T07:35:57.910 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:35:58.812 INFO:tasks.workunit.client.0.smithi134.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:35:58.812 INFO:tasks.workunit.client.0.smithi134.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:35:58.812 INFO:tasks.workunit.client.0.smithi134.stdout:2 : device size 0x101000000 : using 0x814b1000(2.0 GiB) 2024-04-25T07:35:58.812 INFO:tasks.workunit.client.0.smithi134.stdout:Expanding DB/WAL... 2024-04-25T07:35:58.812 INFO:tasks.workunit.client.0.smithi134.stdout:0 : expanding to 0x536870912 2024-04-25T07:35:58.857 INFO:tasks.workunit.client.0.smithi134.stdout:0 : size label updated to 536870912 2024-04-25T07:35:58.857 INFO:tasks.workunit.client.0.smithi134.stdout:1 : expanding to 0x1073741824 2024-04-25T07:35:58.907 INFO:tasks.workunit.client.0.smithi134.stdout:1 : size label updated to 1073741824 2024-04-25T07:35:58.907 INFO:tasks.workunit.client.0.smithi134.stdout:2 : expanding from 0x80000000 to 0x101000000 2024-04-25T07:35:59.125 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:35:59.128+0000 7f786e421ac0 -1 bluestore(td/osd-bluefs-volume-ops/1) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-04-25T07:35:59.125 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:35:59.128+0000 7f786e421ac0 -1 bluestore(td/osd-bluefs-volume-ops/1) unable to read label for td/osd-bluefs-volume-ops/1: (21) Is a directory 2024-04-25T07:36:04.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:77: TEST_bluestore: truncate td/osd-bluefs-volume-ops/2/block -s 4295099392 2024-04-25T07:36:04.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:78: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 bluefs-bdev-expand 2024-04-25T07:36:04.434 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:05.338 INFO:tasks.workunit.client.0.smithi134.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:36:05.338 INFO:tasks.workunit.client.0.smithi134.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:36:05.338 INFO:tasks.workunit.client.0.smithi134.stdout:2 : device size 0x100020000 : using 0x804ce000(2.0 GiB) 2024-04-25T07:36:05.339 INFO:tasks.workunit.client.0.smithi134.stdout:Expanding DB/WAL... 2024-04-25T07:36:05.339 INFO:tasks.workunit.client.0.smithi134.stdout:0 : expanding to 0x536870912 2024-04-25T07:36:05.372 INFO:tasks.workunit.client.0.smithi134.stdout:0 : size label updated to 536870912 2024-04-25T07:36:05.372 INFO:tasks.workunit.client.0.smithi134.stdout:1 : expanding to 0x1073741824 2024-04-25T07:36:05.405 INFO:tasks.workunit.client.0.smithi134.stdout:1 : size label updated to 1073741824 2024-04-25T07:36:05.405 INFO:tasks.workunit.client.0.smithi134.stdout:2 : expanding from 0x80000000 to 0x100020000 2024-04-25T07:36:05.648 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:05.652+0000 7f3463d4aac0 -1 bluestore(td/osd-bluefs-volume-ops/2) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-04-25T07:36:05.648 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:05.652+0000 7f3463d4aac0 -1 bluestore(td/osd-bluefs-volume-ops/2) unable to read label for td/osd-bluefs-volume-ops/2: (21) Is a directory 2024-04-25T07:36:10.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:79: TEST_bluestore: truncate td/osd-bluefs-volume-ops/3/block -s 4293918720 2024-04-25T07:36:10.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:80: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 bluefs-bdev-expand 2024-04-25T07:36:10.754 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:11.645 INFO:tasks.workunit.client.0.smithi134.stdout:0 : device size 0x1ffff000 : using 0x3a00000(58 MiB) 2024-04-25T07:36:11.645 INFO:tasks.workunit.client.0.smithi134.stdout:1 : device size 0x3fffe000 : using 0xd00000(13 MiB) 2024-04-25T07:36:11.646 INFO:tasks.workunit.client.0.smithi134.stdout:2 : device size 0xfff00000 : using 0x803a5000(2.0 GiB) 2024-04-25T07:36:11.646 INFO:tasks.workunit.client.0.smithi134.stdout:Expanding DB/WAL... 2024-04-25T07:36:11.646 INFO:tasks.workunit.client.0.smithi134.stdout:0 : expanding to 0x536870912 2024-04-25T07:36:11.679 INFO:tasks.workunit.client.0.smithi134.stdout:0 : size label updated to 536870912 2024-04-25T07:36:11.679 INFO:tasks.workunit.client.0.smithi134.stdout:1 : expanding to 0x1073741824 2024-04-25T07:36:11.712 INFO:tasks.workunit.client.0.smithi134.stdout:1 : size label updated to 1073741824 2024-04-25T07:36:11.712 INFO:tasks.workunit.client.0.smithi134.stdout:2 : expanding from 0x80000000 to 0xfff00000 2024-04-25T07:36:11.922 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:11.924+0000 7f49ff688ac0 -1 bluestore(td/osd-bluefs-volume-ops/3) _read_bdev_label failed to read from td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-04-25T07:36:11.922 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:11.924+0000 7f49ff688ac0 -1 bluestore(td/osd-bluefs-volume-ops/3) unable to read label for td/osd-bluefs-volume-ops/3: (21) Is a directory 2024-04-25T07:36:17.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:83: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:36:19.073 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:19.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:84: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:36:20.901 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:20.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:85: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:36:22.733 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:22.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:86: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:36:24.565 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:24.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:88: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-04-25T07:36:24.612 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:26.394 INFO:tasks.workunit.client.0.smithi134.stdout:0 : device size 0x1ffff000 : using 0x1600000(22 MiB) 2024-04-25T07:36:26.395 INFO:tasks.workunit.client.0.smithi134.stdout:1 : device size 0x3fffe000 : using 0x1600000(22 MiB) 2024-04-25T07:36:26.395 INFO:tasks.workunit.client.0.smithi134.stdout:2 : device size 0x100000000 : using 0x4a0000(4.6 MiB) 2024-04-25T07:36:26.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:90: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/block.db --command bluefs-bdev-migrate 2024-04-25T07:36:26.424 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:28.205 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-04-25T07:36:28.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:95: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:36:29.525 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:29.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:98: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-04-25T07:36:29.570 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:31.601 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:36:31.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:103: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:36:32.917 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:32.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:106: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block.wal --devs-source td/osd-bluefs-volume-ops/2/block.db --dev-target td/osd-bluefs-volume-ops/2/block --command bluefs-bdev-migrate 2024-04-25T07:36:32.963 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:34.749 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-04-25T07:36:34.749 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-04-25T07:36:34.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:112: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:36:35.565 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:35.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:115: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block.wal --command bluefs-bdev-migrate 2024-04-25T07:36:35.604 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:Migrate to WAL device isn't supported. 2024-04-25T07:36:35.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:121: TEST_bluestore: '[' 1 -eq 0 ']' 2024-04-25T07:36:35.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:124: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:36:37.409 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:37.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:127: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-04-25T07:36:37.455 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:39.489 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-04-25T07:36:39.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:132: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:36:40.829 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:40.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:134: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --dev-target td/osd-bluefs-volume-ops/3/block.db --command bluefs-bdev-migrate 2024-04-25T07:36:40.878 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:36:42.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:139: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:36:43.481 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:36:43.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:141: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:36:43.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:36:43.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:36:43.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:36:43.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:36:43.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:36:43.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:36:43.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:43.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:43.498 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:43.498 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:43.498 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:36:43.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:36:43.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:36:43.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:36:43.500 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:36:43.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:36:43.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:36:43.501 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:36:43.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:36:43.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:36:43.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:36:43.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:36:43.536 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:43.535+0000 7f0d975007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:43.539 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:43.539+0000 7f0d975007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:43.540 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:43.539+0000 7f0d975007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:43.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:36:43.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:36:43.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:36:43.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:36:43.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:36:43.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:43.904 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:36:43.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:36:43.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:43.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:44.323 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:44.323+0000 7f0d975007c0 -1 Falling back to public interface 2024-04-25T07:36:44.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:45.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:45.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:45.343 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:36:45.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:36:45.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:45.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:45.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:46.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:46.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:46.745 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:36:46.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:36:46.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:46.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:47.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:48.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:48.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:48.149 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:36:48.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:36:48.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:48.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:48.237 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:48.238+0000 7f0d975007c0 -1 osd.0 27 log_to_monitors true 2024-04-25T07:36:48.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:49.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:49.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:49.667 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:36:49.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:36:49.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:49.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:36:50.130 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 38 up_thru 38 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/3565070792,v1:127.0.0.1:6803/3565070792] [v2:127.0.0.1:6804/3565070792,v1:127.0.0.1:6805/3565070792] exists,up a2d96bda-b606-4fae-8541-2faf9c7a0ded 2024-04-25T07:36:50.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:36:50.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:36:50.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:36:50.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:36:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:142: TEST_bluestore: osd_pid0=117757 2024-04-25T07:36:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:143: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:36:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:36:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:36:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:36:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:36:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:36:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:36:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:36:50.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:36:50.137 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:36:50.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:36:50.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:36:50.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:36:50.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:36:50.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:36:50.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:36:50.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:36:50.161 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:50.162+0000 7fc52aa4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:50.164 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:50.162+0000 7fc52aa4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:50.165 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:50.166+0000 7fc52aa4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:36:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:36:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:36:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:36:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:36:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:50.581 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:36:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:36:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:36:50.971 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:50.970+0000 7fc52aa4f7c0 -1 Falling back to public interface 2024-04-25T07:36:51.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:52.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:52.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:52.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:36:52.003 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:36:52.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:52.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:36:52.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:53.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:53.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:53.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:36:53.428 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:36:53.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:53.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:36:53.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:54.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:54.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:54.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:36:54.830 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:36:54.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:54.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:36:54.840 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:54.838+0000 7fc52aa4f7c0 -1 osd.1 30 log_to_monitors true 2024-04-25T07:36:55.310 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 41 up_thru 28 down_at 31 last_clean_interval [10,30) [v2:127.0.0.1:6810/2916832177,v1:127.0.0.1:6811/2916832177] [v2:127.0.0.1:6812/2916832177,v1:127.0.0.1:6813/2916832177] exists,up ac851636-c635-4577-a7fd-b89070ff1f35 2024-04-25T07:36:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:36:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:36:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:36:55.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:36:55.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:144: TEST_bluestore: osd_pid1=118721 2024-04-25T07:36:55.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:145: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:36:55.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:36:55.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:36:55.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:36:55.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:36:55.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:36:55.316 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:36:55.316 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:36:55.316 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:36:55.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:36:55.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:36:55.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:36:55.318 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:36:55.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:36:55.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:36:55.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:36:55.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:36:55.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:36:55.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:36:55.342 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:55.342+0000 7fd251a037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:55.344 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:55.342+0000 7fd251a037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:55.345 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:55.346+0000 7fd251a037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:36:55.887 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:55.886+0000 7fd251a037c0 -1 Falling back to public interface 2024-04-25T07:36:55.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:36:55.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:36:55.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:36:55.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:36:55.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:36:55.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:55.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:36:55.890 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:36:55.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:55.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:36:56.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:57.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:57.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:57.588 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:36:57.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:36:57.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:57.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:36:57.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:36:58.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:36:58.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:36:58.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:36:58.986 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:36:58.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:36:58.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:36:59.077 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:36:59.078+0000 7fd251a037c0 -1 osd.2 33 log_to_monitors true 2024-04-25T07:36:59.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:00.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:00.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:00.480 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:37:00.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:37:00.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:00.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:37:01.157 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 45 up_thru 45 down_at 34 last_clean_interval [15,33) [v2:127.0.0.1:6818/3951025535,v1:127.0.0.1:6819/3951025535] [v2:127.0.0.1:6820/3951025535,v1:127.0.0.1:6821/3951025535] exists,up 2febbe7e-a53e-4e2d-9cb9-0f565790e7f7 2024-04-25T07:37:01.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:01.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:01.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:01.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:37:01.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:146: TEST_bluestore: osd_pid2=119660 2024-04-25T07:37:01.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:147: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:37:01.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:37:01.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:37:01.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:37:01.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:37:01.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:37:01.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:37:01.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:37:01.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:37:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:37:01.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:37:01.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:37:01.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:37:01.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:37:01.167 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T07:37:01.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:37:01.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:37:01.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:37:01.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:37:01.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:37:01.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:37:01.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:37:01.190 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:37:01.190+0000 7fca18b2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:01.193 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:37:01.190+0000 7fca18b2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:01.194 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:37:01.194+0000 7fca18b2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:37:01.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:37:01.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:37:01.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:37:01.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:37:01.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:37:01.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:01.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:37:01.666 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:37:01.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:01.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:01.979 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:37:01.978+0000 7fca18b2f7c0 -1 Falling back to public interface 2024-04-25T07:37:02.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:03.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:03.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:03.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:37:03.104 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:37:03.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:03.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:03.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:04.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:04.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:04.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:37:04.519 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:37:04.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:04.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:04.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:05.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:05.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:05.932 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:37:05.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:37:05.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:05.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:06.075 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:37:06.074+0000 7fca18b2f7c0 -1 osd.3 35 log_to_monitors true 2024-04-25T07:37:06.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:37:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:37:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:37:07.504 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:37:07.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:37:07.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:37:07.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:37:08.134 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 49 up_thru 49 down_at 36 last_clean_interval [20,35) [v2:127.0.0.1:6826/2826709558,v1:127.0.0.1:6827/2826709558] [v2:127.0.0.1:6828/2826709558,v1:127.0.0.1:6829/2826709558] exists,up b4287a97-f3f4-4057-a2f6-cd858e9a523b 2024-04-25T07:37:08.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:37:08.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:37:08.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:37:08.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:37:08.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:148: TEST_bluestore: osd_pid3=120594 2024-04-25T07:37:08.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:150: TEST_bluestore: wait_for_clean 2024-04-25T07:37:08.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:37:08.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:37:08.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:37:08.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:37:08.138 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:37:08.138 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:37:08.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:37:08.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:37:08.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:37:08.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:37:08.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:37:08.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:37:08.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:37:08.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:37:08.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:37:08.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:37:08.831 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:37:08.831 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T07:37:08.831 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T07:37:08.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:37:08.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:37:08.832 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:37:08.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=163208757255 2024-04-25T07:37:08.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 163208757255 2024-04-25T07:37:08.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-163208757255' 2024-04-25T07:37:08.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:37:08.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:37:09.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=176093659141 2024-04-25T07:37:09.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 176093659141 2024-04-25T07:37:09.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-163208757255 1-176093659141' 2024-04-25T07:37:09.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:37:09.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:37:09.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=193273528324 2024-04-25T07:37:09.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 193273528324 2024-04-25T07:37:09.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324' 2024-04-25T07:37:09.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:37:09.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:37:09.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=210453397507 2024-04-25T07:37:09.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 210453397507 2024-04-25T07:37:09.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-163208757255 1-176093659141 2-193273528324 3-210453397507' 2024-04-25T07:37:09.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:37:09.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:37:09.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-163208757255 2024-04-25T07:37:09.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:37:09.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-163208757255 2024-04-25T07:37:09.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:37:09.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=163208757255 2024-04-25T07:37:09.456 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 163208757255 2024-04-25T07:37:09.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 163208757255' 2024-04-25T07:37:09.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:37:09.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757252 -lt 163208757255 2024-04-25T07:37:09.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:37:10.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:37:10.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:37:11.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 163208757255 -lt 163208757255 2024-04-25T07:37:11.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:37:11.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:37:11.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-176093659141 2024-04-25T07:37:11.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:37:11.297 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-176093659141 2024-04-25T07:37:11.297 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:37:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=176093659141 2024-04-25T07:37:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 176093659141' 2024-04-25T07:37:11.299 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 176093659141 2024-04-25T07:37:11.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:37:11.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 176093659142 -lt 176093659141 2024-04-25T07:37:11.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:37:11.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-193273528324 2024-04-25T07:37:11.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:37:11.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:37:11.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-193273528324 2024-04-25T07:37:11.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:37:11.743 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 193273528324 2024-04-25T07:37:11.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=193273528324 2024-04-25T07:37:11.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 193273528324' 2024-04-25T07:37:11.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:37:12.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 193273528325 -lt 193273528324 2024-04-25T07:37:12.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:37:12.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-210453397507 2024-04-25T07:37:12.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:37:12.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:37:12.153 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-210453397507 2024-04-25T07:37:12.153 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:37:12.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=210453397507 2024-04-25T07:37:12.154 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 210453397507 2024-04-25T07:37:12.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 210453397507' 2024-04-25T07:37:12.154 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:37:12.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 210453397508 -lt 210453397507 2024-04-25T07:37:12.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:37:12.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:37:12.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:37:13.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:37:13.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:37:13.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:37:13.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:37:13.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:37:13.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:37:13.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:37:13.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:37:13.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:37:13.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:37:13.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:37:13.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:37:14.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:37:14.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:37:14.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:37:14.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:153: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:37:14.158 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T07:37:14.159 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:37:14.159 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_121926 2024-04-25T07:37:14.159 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:37:14.159 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:37:15.159 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 58 42 0.164051 0.164062 0.278861 0.307808 2024-04-25T07:37:16.159 INFO:tasks.workunit.client.0.smithi134.stdout: 2 16 111 95 0.185527 0.207031 0.434001 0.315829 2024-04-25T07:37:17.159 INFO:tasks.workunit.client.0.smithi134.stdout: 3 16 154 138 0.179665 0.167969 0.389846 0.324622 2024-04-25T07:37:18.160 INFO:tasks.workunit.client.0.smithi134.stdout: 4 16 205 189 0.184547 0.199219 0.33055 0.328745 2024-04-25T07:37:19.159 INFO:tasks.workunit.client.0.smithi134.stdout: 5 16 257 241 0.188255 0.203125 0.247242 0.325099 2024-04-25T07:37:20.160 INFO:tasks.workunit.client.0.smithi134.stdout: 6 16 314 298 0.193982 0.222656 0.144353 0.313439 2024-04-25T07:37:21.160 INFO:tasks.workunit.client.0.smithi134.stdout: 7 16 364 348 0.194168 0.195312 0.179037 0.311134 2024-04-25T07:37:22.160 INFO:tasks.workunit.client.0.smithi134.stdout: 8 16 415 399 0.194795 0.199219 0.381955 0.312767 2024-04-25T07:37:23.160 INFO:tasks.workunit.client.0.smithi134.stdout: 9 16 461 445 0.193113 0.179688 0.223944 0.315863 2024-04-25T07:37:24.160 INFO:tasks.workunit.client.0.smithi134.stdout: 10 16 515 499 0.194894 0.210938 0.232866 0.315962 2024-04-25T07:37:25.160 INFO:tasks.workunit.client.0.smithi134.stdout: 11 16 570 554 0.196704 0.214844 0.28653 0.314411 2024-04-25T07:37:26.161 INFO:tasks.workunit.client.0.smithi134.stdout: 12 16 621 605 0.196912 0.199219 0.323488 0.315368 2024-04-25T07:37:27.161 INFO:tasks.workunit.client.0.smithi134.stdout: 13 16 672 656 0.197087 0.199219 0.252699 0.313092 2024-04-25T07:37:28.162 INFO:tasks.workunit.client.0.smithi134.stdout: 14 16 717 701 0.195563 0.175781 0.549754 0.315764 2024-04-25T07:37:29.161 INFO:tasks.workunit.client.0.smithi134.stdout: 15 16 770 754 0.196326 0.207031 0.212273 0.315839 2024-04-25T07:37:30.161 INFO:tasks.workunit.client.0.smithi134.stdout: 16 16 822 806 0.19675 0.203125 0.179892 0.315206 2024-04-25T07:37:31.161 INFO:tasks.workunit.client.0.smithi134.stdout: 17 16 869 853 0.195974 0.183594 0.273816 0.31522 2024-04-25T07:37:32.161 INFO:tasks.workunit.client.0.smithi134.stdout: 18 16 925 909 0.197238 0.21875 0.307034 0.313898 2024-04-25T07:37:33.162 INFO:tasks.workunit.client.0.smithi134.stdout: 19 16 978 962 0.197751 0.207031 0.26856 0.312217 2024-04-25T07:37:34.162 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:37:34.166796+0000 min lat: 0.144353 max lat: 0.549754 avg lat: 0.311014 2024-04-25T07:37:34.162 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:37:34.162 INFO:tasks.workunit.client.0.smithi134.stdout: 20 16 1039 1023 0.199777 0.238281 0.188563 0.311014 2024-04-25T07:37:35.162 INFO:tasks.workunit.client.0.smithi134.stdout: 21 16 1099 1083 0.201422 0.234375 0.345052 0.308063 2024-04-25T07:37:36.162 INFO:tasks.workunit.client.0.smithi134.stdout: 22 16 1167 1151 0.204339 0.265625 0.208449 0.303979 2024-04-25T07:37:37.162 INFO:tasks.workunit.client.0.smithi134.stdout: 23 16 1240 1224 0.207851 0.285156 0.319014 0.299177 2024-04-25T07:37:38.163 INFO:tasks.workunit.client.0.smithi134.stdout: 24 16 1305 1289 0.209768 0.253906 0.21038 0.296489 2024-04-25T07:37:39.163 INFO:tasks.workunit.client.0.smithi134.stdout: 25 16 1366 1350 0.210907 0.238281 0.267481 0.295049 2024-04-25T07:37:40.163 INFO:tasks.workunit.client.0.smithi134.stdout: 26 16 1432 1416 0.21271 0.257812 0.207859 0.292724 2024-04-25T07:37:41.163 INFO:tasks.workunit.client.0.smithi134.stdout: 27 16 1487 1471 0.212788 0.214844 0.2944 0.292144 2024-04-25T07:37:42.163 INFO:tasks.workunit.client.0.smithi134.stdout: 28 16 1551 1535 0.214115 0.25 0.277526 0.290565 2024-04-25T07:37:43.163 INFO:tasks.workunit.client.0.smithi134.stdout: 29 16 1609 1593 0.214544 0.226562 0.318388 0.289741 2024-04-25T07:37:44.163 INFO:tasks.workunit.client.0.smithi134.stdout: 30 16 1673 1657 0.215725 0.25 0.233958 0.288805 2024-04-25T07:37:45.163 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 30.1525 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 1673 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 4096 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 4096 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 0.216737 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 0.0295839 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 0.285156 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 0.164062 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 55 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 7.57347 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 73 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 42 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 0.288021 2024-04-25T07:37:45.164 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.0654015 2024-04-25T07:37:45.165 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 0.549754 2024-04-25T07:37:45.165 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.134857 2024-04-25T07:37:45.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 117757 2024-04-25T07:37:45.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: sleep 1 2024-04-25T07:37:46.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:156: TEST_bluestore: kill 117757 2024-04-25T07:37:46.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 156: kill: (117757) - No such process 2024-04-25T07:37:46.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:157: TEST_bluestore: ceph osd down 0 2024-04-25T07:37:46.836 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:37:46.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 118721 2024-04-25T07:37:46.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: sleep 1 2024-04-25T07:37:47.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:158: TEST_bluestore: kill 118721 2024-04-25T07:37:47.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 158: kill: (118721) - No such process 2024-04-25T07:37:47.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:159: TEST_bluestore: ceph osd down 1 2024-04-25T07:37:48.335 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already down. 2024-04-25T07:37:48.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 119660 2024-04-25T07:37:48.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: sleep 1 2024-04-25T07:37:49.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:160: TEST_bluestore: kill 119660 2024-04-25T07:37:49.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 160: kill: (119660) - No such process 2024-04-25T07:37:49.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:161: TEST_bluestore: ceph osd down 2 2024-04-25T07:37:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already down. 2024-04-25T07:37:49.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 120594 2024-04-25T07:37:49.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: sleep 1 2024-04-25T07:37:50.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:162: TEST_bluestore: kill 120594 2024-04-25T07:37:50.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 162: kill: (120594) - No such process 2024-04-25T07:37:50.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:163: TEST_bluestore: ceph osd down 3 2024-04-25T07:37:51.351 INFO:tasks.workunit.client.0.smithi134.stderr:osd.3 is already down. 2024-04-25T07:37:51.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:166: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:37:52.673 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:37:52.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:168: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal count=512 bs=1M 2024-04-25T07:37:53.003 INFO:tasks.workunit.client.0.smithi134.stderr:512+0 records in 2024-04-25T07:37:53.003 INFO:tasks.workunit.client.0.smithi134.stderr:512+0 records out 2024-04-25T07:37:53.004 INFO:tasks.workunit.client.0.smithi134.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.306817 s, 1.7 GB/s 2024-04-25T07:37:53.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:169: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --dev-target td/osd-bluefs-volume-ops/0/wal --command bluefs-bdev-new-wal 2024-04-25T07:37:53.027 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:37:57.813 INFO:tasks.workunit.client.0.smithi134.stdout:WAL device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal 2024-04-25T07:37:57.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:173: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:37:59.629 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:37:59.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:176: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:38:00.957 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:38:00.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:178: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db count=1024 bs=1M 2024-04-25T07:38:01.539 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records in 2024-04-25T07:38:01.539 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records out 2024-04-25T07:38:01.539 INFO:tasks.workunit.client.0.smithi134.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.564795 s, 1.9 GB/s 2024-04-25T07:38:01.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:179: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --dev-target td/osd-bluefs-volume-ops/1/db --command bluefs-bdev-new-db 2024-04-25T07:38:01.562 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:38:09.597 INFO:tasks.workunit.client.0.smithi134.stdout:DB device added /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db 2024-04-25T07:38:09.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:183: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block --dev-target td/osd-bluefs-volume-ops/1/block.db --command bluefs-bdev-migrate 2024-04-25T07:38:09.630 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:38:11.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:188: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:38:13.477 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:38:13.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:191: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:38:14.317 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:38:14.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:193: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-db 2024-04-25T07:38:14.352 INFO:tasks.workunit.client.0.smithi134.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:38:14.352 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:38:15.137 INFO:tasks.workunit.client.0.smithi134.stdout:DB device added 2024-04-25T07:38:15.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:196: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --command bluefs-bdev-new-wal 2024-04-25T07:38:15.168 INFO:tasks.workunit.client.0.smithi134.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:38:15.168 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:38:16.449 INFO:tasks.workunit.client.0.smithi134.stdout:WAL device added 2024-04-25T07:38:16.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:199: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/block.db --command bluefs-bdev-migrate 2024-04-25T07:38:16.481 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:38:18.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:204: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:38:20.337 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:38:20.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:207: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:38:21.665 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:38:21.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:209: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --command bluefs-bdev-new-wal 2024-04-25T07:38:21.705 INFO:tasks.workunit.client.0.smithi134.stdout:NOTICE: --dev-target option omitted, will allocate as a file 2024-04-25T07:38:21.705 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:38:22.997 INFO:tasks.workunit.client.0.smithi134.stdout:WAL device added 2024-04-25T07:38:23.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:212: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block.db --dev-target td/osd-bluefs-volume-ops/3/block --command bluefs-bdev-migrate 2024-04-25T07:38:23.032 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:38:25.077 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block.db 2024-04-25T07:38:25.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:217: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:38:26.405 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:38:26.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:219: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:38:26.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:26.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:38:26.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:38:26.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:38:26.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:38:26.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:38:26.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:38:26.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:38:26.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:38:26.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:38:26.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:38:26.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:38:26.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:38:26.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:38:26.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:38:26.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:38:26.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:38:26.425 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:38:26.425 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:26.425 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:38:26.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:38:26.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:38:26.427 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:38:26.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:38:26.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:38:26.428 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:38:26.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:38:26.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:38:26.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:38:26.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:38:26.460 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:26.460+0000 7f9d5f2277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:26.463 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:26.464+0000 7f9d5f2277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:26.464 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:26.464+0000 7f9d5f2277c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:26.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:38:26.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:38:26.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:38:26.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:38:26.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:38:26.895 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:38:26.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:26.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:38:26.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:26.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:27.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:27.500 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:27.500+0000 7f9d5f2277c0 -1 Falling back to public interface 2024-04-25T07:38:28.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:28.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:28.332 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:38:28.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:38:28.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:28.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:28.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:29.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:29.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:29.733 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:38:29.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:38:29.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:29.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:30.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:31.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:31.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:31.152 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:38:31.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:38:31.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:31.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:31.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:32.377 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:32.376+0000 7f9d5f2277c0 -1 osd.0 51 log_to_monitors true 2024-04-25T07:38:32.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:32.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:32.550 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:38:32.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:38:32.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:32.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:38:33.074 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 62 up_thru 62 down_at 52 last_clean_interval [38,51) [v2:127.0.0.1:6802/1710917738,v1:127.0.0.1:6803/1710917738] [v2:127.0.0.1:6804/1710917738,v1:127.0.0.1:6805/1710917738] exists,up a2d96bda-b606-4fae-8541-2faf9c7a0ded 2024-04-25T07:38:33.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:220: TEST_bluestore: osd_pid0=128034 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:221: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:38:33.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:38:33.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:38:33.138 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:38:33.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:38:33.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:38:33.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:38:33.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:38:33.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:38:33.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:38:33.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:38:33.140 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:33.116+0000 7f84581ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:33.202 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:33.204+0000 7f84581ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:33.203 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:33.204+0000 7f84581ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:33.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:38:33.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:38:33.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:38:33.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:38:33.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:38:33.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:33.586 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:38:33.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:38:33.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:33.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:38:34.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:34.256 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:34.256+0000 7f84581ab7c0 -1 Falling back to public interface 2024-04-25T07:38:35.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:35.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:35.015 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:38:35.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:38:35.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:35.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:38:35.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:36.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:36.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:36.457 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:38:36.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:38:36.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:36.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:38:36.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:37.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:37.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:37.902 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:38:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:38:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:38:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:38.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:38.944 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:38.943+0000 7f84581ab7c0 -1 osd.1 54 log_to_monitors true 2024-04-25T07:38:39.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:39.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:39.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:38:39.421 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:38:39.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:39.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:38:40.095 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 65 up_thru 65 down_at 55 last_clean_interval [41,54) [v2:127.0.0.1:6810/2855168604,v1:127.0.0.1:6811/2855168604] [v2:127.0.0.1:6812/2855168604,v1:127.0.0.1:6813/2855168604] exists,up ac851636-c635-4577-a7fd-b89070ff1f35 2024-04-25T07:38:40.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:38:40.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:38:40.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:38:40.096 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:38:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:222: TEST_bluestore: osd_pid1=129021 2024-04-25T07:38:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:223: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:38:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:38:40.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:38:40.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:38:40.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:38:40.101 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:38:40.101 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:40.101 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:38:40.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:38:40.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:38:40.103 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:38:40.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:38:40.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:38:40.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:38:40.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:38:40.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:38:40.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:38:40.133 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:40.135+0000 7faca4b8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:40.135 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:40.135+0000 7faca4b8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:40.137 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:40.135+0000 7faca4b8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:40.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:38:40.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:38:40.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:38:40.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:38:40.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:38:40.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:40.563 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:38:40.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:38:40.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:40.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:41.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:41.176 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:41.175+0000 7faca4b8b7c0 -1 Falling back to public interface 2024-04-25T07:38:42.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:42.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:42.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:38:42.023 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:38:42.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:42.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:42.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:43.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:43.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:43.449 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:38:43.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:38:43.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:43.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:43.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:44.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:44.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:44.855 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:38:44.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:38:44.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:44.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:45.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:46.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:46.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:46.262 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:38:46.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:38:46.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:46.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:46.489 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:46.487+0000 7faca4b8b7c0 -1 osd.2 57 log_to_monitors true 2024-04-25T07:38:46.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:47.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:47.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:47.703 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:38:47.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:38:47.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:47.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:38:48.280 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 69 up_thru 55 down_at 58 last_clean_interval [45,57) [v2:127.0.0.1:6818/1275092789,v1:127.0.0.1:6819/1275092789] [v2:127.0.0.1:6820/1275092789,v1:127.0.0.1:6821/1275092789] exists,up 2febbe7e-a53e-4e2d-9cb9-0f565790e7f7 2024-04-25T07:38:48.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:38:48.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:38:48.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:38:48.282 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:38:48.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:224: TEST_bluestore: osd_pid2=129973 2024-04-25T07:38:48.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:225: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:38:48.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:38:48.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:38:48.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:38:48.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:38:48.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:38:48.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:38:48.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:38:48.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:38:48.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:38:48.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:38:48.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:38:48.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:38:48.292 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T07:38:48.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:38:48.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:38:48.293 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:38:48.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:38:48.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:38:48.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:38:48.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:38:48.322 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:48.323+0000 7f485597a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:48.325 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:48.323+0000 7f485597a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:48.326 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:48.327+0000 7f485597a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:38:49.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:38:49.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:38:49.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:38:49.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:38:49.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:38:49.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:49.080 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:38:49.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:38:49.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:49.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:49.139 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:49.139+0000 7f485597a7c0 -1 Falling back to public interface 2024-04-25T07:38:49.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:50.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:50.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:50.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:38:50.474 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:38:50.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:50.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:50.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:51.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:51.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:51.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:38:51.974 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:38:51.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:51.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:52.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:53.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:53.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:53.375 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:38:53.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:38:53.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:53.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:53.599 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:38:53.599+0000 7f485597a7c0 -1 osd.3 59 log_to_monitors true 2024-04-25T07:38:53.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:38:54.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:38:54.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:38:54.813 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:38:54.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:38:54.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:38:54.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:38:55.396 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 74 up_thru 74 down_at 60 last_clean_interval [49,59) [v2:127.0.0.1:6826/2631465033,v1:127.0.0.1:6827/2631465033] [v2:127.0.0.1:6828/2631465033,v1:127.0.0.1:6829/2631465033] exists,up b4287a97-f3f4-4057-a2f6-cd858e9a523b 2024-04-25T07:38:55.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:38:55.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:38:55.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:38:55.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:38:55.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:226: TEST_bluestore: osd_pid3=130967 2024-04-25T07:38:55.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:229: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:38:55.435 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T07:38:55.435 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:38:55.435 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_131916 2024-04-25T07:38:55.435 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:38:55.435 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:38:56.435 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 40 24 0.0937391 0.09375 0.361452 0.544196 2024-04-25T07:38:57.435 INFO:tasks.workunit.client.0.smithi134.stdout: 2 16 88 72 0.14061 0.1875 0.340314 0.423146 2024-04-25T07:38:58.435 INFO:tasks.workunit.client.0.smithi134.stdout: 3 16 127 111 0.144516 0.152344 0.376163 0.395 2024-04-25T07:38:59.435 INFO:tasks.workunit.client.0.smithi134.stdout: 4 16 180 164 0.16014 0.207031 0.343657 0.381245 2024-04-25T07:39:00.435 INFO:tasks.workunit.client.0.smithi134.stdout: 5 16 222 206 0.16092 0.164062 0.38458 0.370324 2024-04-25T07:39:01.436 INFO:tasks.workunit.client.0.smithi134.stdout: 6 16 266 250 0.162742 0.171875 0.343766 0.376427 2024-04-25T07:39:02.436 INFO:tasks.workunit.client.0.smithi134.stdout: 7 16 307 291 0.162368 0.160156 0.259845 0.373435 2024-04-25T07:39:03.436 INFO:tasks.workunit.client.0.smithi134.stdout: 8 16 356 340 0.165996 0.191406 0.199325 0.369495 2024-04-25T07:39:04.436 INFO:tasks.workunit.client.0.smithi134.stdout: 9 16 403 387 0.167948 0.183594 0.253385 0.365574 2024-04-25T07:39:05.436 INFO:tasks.workunit.client.0.smithi134.stdout: 10 16 453 437 0.170683 0.195312 0.35025 0.361996 2024-04-25T07:39:06.436 INFO:tasks.workunit.client.0.smithi134.stdout: 11 16 487 471 0.167238 0.132812 0.354667 0.362835 2024-04-25T07:39:07.436 INFO:tasks.workunit.client.0.smithi134.stdout: 12 16 530 514 0.167297 0.167969 0.229524 0.368227 2024-04-25T07:39:08.437 INFO:tasks.workunit.client.0.smithi134.stdout: 13 16 573 557 0.167346 0.167969 0.257149 0.367997 2024-04-25T07:39:09.437 INFO:tasks.workunit.client.0.smithi134.stdout: 14 16 615 599 0.167109 0.164062 0.352827 0.367616 2024-04-25T07:39:10.437 INFO:tasks.workunit.client.0.smithi134.stdout: 15 16 659 643 0.167425 0.171875 0.4092 0.367595 2024-04-25T07:39:11.437 INFO:tasks.workunit.client.0.smithi134.stdout: 16 16 705 689 0.16819 0.179688 0.34423 0.367909 2024-04-25T07:39:12.437 INFO:tasks.workunit.client.0.smithi134.stdout: 17 16 746 730 0.167716 0.160156 0.384699 0.368047 2024-04-25T07:39:13.437 INFO:tasks.workunit.client.0.smithi134.stdout: 18 16 793 777 0.168597 0.183594 0.35092 0.366121 2024-04-25T07:39:14.438 INFO:tasks.workunit.client.0.smithi134.stdout: 19 15 840 825 0.169591 0.1875 0.358736 0.36472 2024-04-25T07:39:15.438 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:39:15.442468+0000 min lat: 0.198759 max lat: 0.814045 avg lat: 0.364275 2024-04-25T07:39:15.438 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:39:15.438 INFO:tasks.workunit.client.0.smithi134.stdout: 20 16 887 871 0.170095 0.179688 0.362834 0.364275 2024-04-25T07:39:16.438 INFO:tasks.workunit.client.0.smithi134.stdout: 21 16 933 917 0.170551 0.179688 0.220917 0.363124 2024-04-25T07:39:17.438 INFO:tasks.workunit.client.0.smithi134.stdout: 22 16 977 961 0.17061 0.171875 0.375415 0.363183 2024-04-25T07:39:18.438 INFO:tasks.workunit.client.0.smithi134.stdout: 23 16 1023 1007 0.171004 0.179688 0.243644 0.362419 2024-04-25T07:39:19.438 INFO:tasks.workunit.client.0.smithi134.stdout: 24 16 1070 1054 0.171527 0.183594 0.242332 0.361834 2024-04-25T07:39:20.438 INFO:tasks.workunit.client.0.smithi134.stdout: 25 16 1120 1104 0.172477 0.195312 0.219161 0.360224 2024-04-25T07:39:21.438 INFO:tasks.workunit.client.0.smithi134.stdout: 26 16 1162 1146 0.172153 0.164062 0.353404 0.360967 2024-04-25T07:39:22.438 INFO:tasks.workunit.client.0.smithi134.stdout: 27 16 1200 1184 0.171274 0.148438 0.480616 0.361035 2024-04-25T07:39:23.438 INFO:tasks.workunit.client.0.smithi134.stdout: 28 16 1251 1235 0.172271 0.199219 0.2214 0.360844 2024-04-25T07:39:24.439 INFO:tasks.workunit.client.0.smithi134.stdout: 29 16 1293 1277 0.171987 0.164062 0.245506 0.360307 2024-04-25T07:39:25.439 INFO:tasks.workunit.client.0.smithi134.stdout: 30 16 1344 1328 0.172894 0.199219 0.206866 0.360031 2024-04-25T07:39:26.439 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 30.2579 2024-04-25T07:39:26.439 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 1344 2024-04-25T07:39:26.439 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 4096 2024-04-25T07:39:26.439 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 4096 2024-04-25T07:39:26.439 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 0.173508 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 0.0223088 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 0.207031 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 0.09375 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 44 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 5.71105 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 53 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 24 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 0.359414 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.0726533 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 0.814045 2024-04-25T07:39:26.440 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.198759 2024-04-25T07:39:26.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 128034 2024-04-25T07:39:26.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: sleep 1 2024-04-25T07:39:27.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:232: TEST_bluestore: kill 128034 2024-04-25T07:39:27.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 232: kill: (128034) - No such process 2024-04-25T07:39:27.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:233: TEST_bluestore: ceph osd down 0 2024-04-25T07:39:28.220 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:39:28.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 129021 2024-04-25T07:39:28.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: sleep 1 2024-04-25T07:39:29.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:234: TEST_bluestore: kill 129021 2024-04-25T07:39:29.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 234: kill: (129021) - No such process 2024-04-25T07:39:29.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:235: TEST_bluestore: ceph osd down 1 2024-04-25T07:39:29.952 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already down. 2024-04-25T07:39:29.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 129973 2024-04-25T07:39:29.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: sleep 1 2024-04-25T07:39:30.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:236: TEST_bluestore: kill 129973 2024-04-25T07:39:30.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 236: kill: (129973) - No such process 2024-04-25T07:39:30.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:237: TEST_bluestore: ceph osd down 2 2024-04-25T07:39:31.381 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already down. 2024-04-25T07:39:31.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 130967 2024-04-25T07:39:31.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: sleep 1 2024-04-25T07:39:32.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:238: TEST_bluestore: kill 130967 2024-04-25T07:39:32.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 238: kill: (130967) - No such process 2024-04-25T07:39:32.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:239: TEST_bluestore: ceph osd down 3 2024-04-25T07:39:32.977 INFO:tasks.workunit.client.0.smithi134.stderr:osd.3 is already down. 2024-04-25T07:39:32.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:242: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:39:34.801 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:39:34.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:244: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/db2 count=1024 bs=1M 2024-04-25T07:39:35.394 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records in 2024-04-25T07:39:35.394 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records out 2024-04-25T07:39:35.395 INFO:tasks.workunit.client.0.smithi134.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.56756 s, 1.9 GB/s 2024-04-25T07:39:35.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:245: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/db2 --command bluefs-bdev-migrate 2024-04-25T07:39:35.419 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:44.705 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-04-25T07:39:44.706 INFO:tasks.workunit.client.0.smithi134.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/db2 2024-04-25T07:39:44.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:250: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:39:46.521 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:39:46.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:254: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/0/wal2 count=512 bs=1M 2024-04-25T07:39:46.843 INFO:tasks.workunit.client.0.smithi134.stderr:512+0 records in 2024-04-25T07:39:46.843 INFO:tasks.workunit.client.0.smithi134.stderr:512+0 records out 2024-04-25T07:39:46.844 INFO:tasks.workunit.client.0.smithi134.stderr:536870912 bytes (537 MB, 512 MiB) copied, 0.301769 s, 1.8 GB/s 2024-04-25T07:39:46.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:255: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.wal --dev-target td/osd-bluefs-volume-ops/0/wal2 --command bluefs-bdev-migrate 2024-04-25T07:39:46.867 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:39:52.401 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:0 td/osd-bluefs-volume-ops/0/block.wal 2024-04-25T07:39:52.401 INFO:tasks.workunit.client.0.smithi134.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/0/wal2 2024-04-25T07:39:52.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:259: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/0/wal 2024-04-25T07:39:52.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:261: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:39:54.289 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:39:54.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:264: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:39:56.117 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:39:56.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:266: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/1/db2 count=1024 bs=1M 2024-04-25T07:39:56.708 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records in 2024-04-25T07:39:56.708 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records out 2024-04-25T07:39:56.709 INFO:tasks.workunit.client.0.smithi134.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.567387 s, 1.9 GB/s 2024-04-25T07:39:56.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:267: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --devs-source td/osd-bluefs-volume-ops/1/block.wal --dev-target td/osd-bluefs-volume-ops/1/db2 --command bluefs-bdev-migrate 2024-04-25T07:39:56.732 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:40:05.766 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:40:05.766 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:0 td/osd-bluefs-volume-ops/1/block.wal 2024-04-25T07:40:05.766 INFO:tasks.workunit.client.0.smithi134.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/1/db2 2024-04-25T07:40:05.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:273: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db 2024-04-25T07:40:05.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:275: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:40:07.213 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:40:07.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:277: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 --devs-source td/osd-bluefs-volume-ops/1/block.db --dev-target td/osd-bluefs-volume-ops/1/block --command bluefs-bdev-migrate 2024-04-25T07:40:07.255 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:40:08.545 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:1 td/osd-bluefs-volume-ops/1/block.db 2024-04-25T07:40:08.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:282: TEST_bluestore: rm -rf td/osd-bluefs-volume-ops/1/db2 2024-04-25T07:40:08.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:284: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/1 fsck 2024-04-25T07:40:09.485 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:40:09.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:287: TEST_bluestore: ceph-objectstore-tool --type bluestore --data-path td/osd-bluefs-volume-ops/2 --op fsck --no-mon-config 2024-04-25T07:40:11.305 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:40:11.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:290: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-04-25T07:40:11.895 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records in 2024-04-25T07:40:11.895 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records out 2024-04-25T07:40:11.896 INFO:tasks.workunit.client.0.smithi134.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.564087 s, 1.9 GB/s 2024-04-25T07:40:11.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:291: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-04-25T07:40:11.918 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:40:11.919 INFO:tasks.workunit.client.0.smithi134.stderr:Unable to migrate Slow volume to new location, please allocate new DB or WAL with --bluefs-bdev-new-db(wal) command 2024-04-25T07:40:11.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:297: TEST_bluestore: '[' 1 -eq 0 ']' 2024-04-25T07:40:11.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:300: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:40:13.725 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:40:13.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:303: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/2/db2 count=1024 bs=1M 2024-04-25T07:40:21.329 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records in 2024-04-25T07:40:21.329 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records out 2024-04-25T07:40:21.329 INFO:tasks.workunit.client.0.smithi134.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 7.44309 s, 144 MB/s 2024-04-25T07:40:21.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:305: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 --devs-source td/osd-bluefs-volume-ops/2/block --devs-source td/osd-bluefs-volume-ops/2/block.db --devs-source td/osd-bluefs-volume-ops/2/block.wal --dev-target td/osd-bluefs-volume-ops/2/db2 --command bluefs-bdev-migrate 2024-04-25T07:40:21.355 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:40:23.889 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:1 td/osd-bluefs-volume-ops/2/block.db 2024-04-25T07:40:23.889 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:0 td/osd-bluefs-volume-ops/2/block.wal 2024-04-25T07:40:23.889 INFO:tasks.workunit.client.0.smithi134.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/2/db2 2024-04-25T07:40:23.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:312: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/2 fsck 2024-04-25T07:40:25.221 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:40:25.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:315: TEST_bluestore: dd if=/dev/zero of=td/osd-bluefs-volume-ops/3/wal2 count=1024 bs=1M 2024-04-25T07:40:25.813 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records in 2024-04-25T07:40:25.813 INFO:tasks.workunit.client.0.smithi134.stderr:1024+0 records out 2024-04-25T07:40:25.813 INFO:tasks.workunit.client.0.smithi134.stderr:1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.574041 s, 1.9 GB/s 2024-04-25T07:40:25.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:317: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 --devs-source td/osd-bluefs-volume-ops/3/block --devs-source td/osd-bluefs-volume-ops/3/block.wal --dev-target td/osd-bluefs-volume-ops/3/wal2 --command bluefs-bdev-migrate 2024-04-25T07:40:25.836 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:40:34.369 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:1 td/osd-bluefs-volume-ops/3/block 2024-04-25T07:40:34.369 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:0 td/osd-bluefs-volume-ops/3/block.wal 2024-04-25T07:40:34.370 INFO:tasks.workunit.client.0.smithi134.stdout: device added: 1 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td/osd-bluefs-volume-ops/3/wal2 2024-04-25T07:40:34.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:323: TEST_bluestore: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/3 fsck 2024-04-25T07:40:35.701 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:325: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:40:35.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:40:35.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:35.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:40:35.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:40:35.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:40:35.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:35.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:35.721 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:35.721 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:35.721 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:40:35.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:40:35.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:40:35.723 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:40:35.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:40:35.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:35.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:40:35.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:40:35.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:40:35.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:40:35.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:40:35.759 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:35.761+0000 7f97246a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:35.762 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:35.765+0000 7f97246a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:35.764 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:35.765+0000 7f97246a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:36.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:40:36.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:36.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:40:36.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:36.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:36.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:36.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:36.152 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:40:36.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:36.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:36.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:36.812 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:36.813+0000 7f97246a57c0 -1 Falling back to public interface 2024-04-25T07:40:37.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:37.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:37.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:37.574 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:40:37.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:37.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:37.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:38.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:38.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:38.983 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:40:38.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:38.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:38.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:39.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:40.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:40.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:40.385 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:40:40.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:40.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:40.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:40.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:41.535 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:41.537+0000 7f97246a57c0 -1 osd.0 76 log_to_monitors true 2024-04-25T07:40:41.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:41.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:41.838 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:40:41.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:40:41.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:41.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:40:42.324 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 87 up_thru 87 down_at 77 last_clean_interval [62,76) [v2:127.0.0.1:6802/3576452767,v1:127.0.0.1:6803/3576452767] [v2:127.0.0.1:6804/3576452767,v1:127.0.0.1:6805/3576452767] exists,up a2d96bda-b606-4fae-8541-2faf9c7a0ded 2024-04-25T07:40:42.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:42.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:42.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:42.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:326: TEST_bluestore: osd_pid0=138046 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:327: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 1 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/1 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/1' 2024-04-25T07:40:42.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/1/journal' 2024-04-25T07:40:42.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:42.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:40:42.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:40:42.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:40:42.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:42.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:42.330 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:40:42.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/1 2024-04-25T07:40:42.332 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:40:42.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:40:42.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/1 --osd-journal=td/osd-bluefs-volume-ops/1/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:42.333 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/1/whoami 2024-04-25T07:40:42.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:40:42.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:40:42.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:40:42.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:40:42.360 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:42.361+0000 7f691483b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:42.379 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:42.381+0000 7f691483b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:42.381 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:42.381+0000 7f691483b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:42.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:40:42.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:42.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:40:42.746 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:40:42.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:42.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:42.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:42.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:42.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:42.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:42.916 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:42.917+0000 7f691483b7c0 -1 Falling back to public interface 2024-04-25T07:40:43.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:44.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:44.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:44.148 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:40:44.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:44.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:44.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:44.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:45.550 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:40:45.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:45.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:45.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:45.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:45.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:45.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:46.374 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:46.377+0000 7f691483b7c0 -1 osd.1 79 log_to_monitors true 2024-04-25T07:40:46.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:46.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:46.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:46.960 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:40:46.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:46.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:40:47.526 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 90 up_thru 77 down_at 80 last_clean_interval [65,79) [v2:127.0.0.1:6810/15005730,v1:127.0.0.1:6811/15005730] [v2:127.0.0.1:6812/15005730,v1:127.0.0.1:6813/15005730] exists,up ac851636-c635-4577-a7fd-b89070ff1f35 2024-04-25T07:40:47.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:47.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:47.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:47.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.1.pid 2024-04-25T07:40:47.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:328: TEST_bluestore: osd_pid1=139015 2024-04-25T07:40:47.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:329: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 2 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/2 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/2' 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/2/journal' 2024-04-25T07:40:47.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:47.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:40:47.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:40:47.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:40:47.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:47.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:47.532 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:47.532 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:40:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/2 2024-04-25T07:40:47.534 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:40:47.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:40:47.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/2 --osd-journal=td/osd-bluefs-volume-ops/2/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:47.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/2/whoami 2024-04-25T07:40:47.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:40:47.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:40:47.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:40:47.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:40:47.566 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:47.569+0000 7efec62457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:47.665 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:47.669+0000 7efec62457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:47.666 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:47.669+0000 7efec62457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:47.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:40:47.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:47.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:40:47.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:47.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:47.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:47.997 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:40:47.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:47.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:47.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:48.451 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:48.453+0000 7efec62457c0 -1 Falling back to public interface 2024-04-25T07:40:48.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:49.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:49.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:49.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:49.455 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:40:49.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:49.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:49.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:50.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:50.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:50.869 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:40:50.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:50.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:50.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:51.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:52.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:52.285 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:40:52.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:52.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:52.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:52.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:52.432 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:52.432+0000 7efec62457c0 -1 osd.2 82 log_to_monitors true 2024-04-25T07:40:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:53.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:53.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:53.747 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:40:53.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:40:53.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:53.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:40:54.257 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 94 up_thru 94 down_at 83 last_clean_interval [69,82) [v2:127.0.0.1:6818/1687824630,v1:127.0.0.1:6819/1687824630] [v2:127.0.0.1:6820/1687824630,v1:127.0.0.1:6821/1687824630] exists,up 2febbe7e-a53e-4e2d-9cb9-0f565790e7f7 2024-04-25T07:40:54.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:40:54.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:40:54.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:40:54.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.2.pid 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:330: TEST_bluestore: osd_pid2=139966 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:331: TEST_bluestore: activate_osd td/osd-bluefs-volume-ops 3 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=3 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/3 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true ' 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:40:54.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:40:54.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:40:54.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/3' 2024-04-25T07:40:54.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/3/journal' 2024-04-25T07:40:54.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:40:54.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:40:54.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:40:54.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:40:54.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:40:54.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:40:54.263 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:40:54.263 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:40:54.263 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:40:54.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:40:54.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/3 2024-04-25T07:40:54.265 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T07:40:54.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.3 2024-04-25T07:40:54.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 3 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/3 --osd-journal=td/osd-bluefs-volume-ops/3/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:40:54.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/3/whoami 2024-04-25T07:40:54.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 3 = 3 ']' 2024-04-25T07:40:54.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:40:54.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:40:54.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:40:54.296 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:54.296+0000 7f816c7667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:54.298 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:54.300+0000 7f816c7667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:54.299 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:54.300+0000 7f816c7667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:40:54.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 3 2024-04-25T07:40:54.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:40:54.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T07:40:54.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:40:54.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:40:54.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:54.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:40:54.722 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:40:54.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:54.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:40:55.083 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:55.084+0000 7f816c7667c0 -1 Falling back to public interface 2024-04-25T07:40:55.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:56.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:56.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:56.231 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:40:56.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:40:56.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:56.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:40:56.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:57.664 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:40:57.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:57.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:57.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:40:57.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:57.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:40:58.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:40:59.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:40:59.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:40:59.073 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:40:59.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:40:59.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:40:59.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:40:59.165 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:40:59.168+0000 7f816c7667c0 -1 osd.3 84 log_to_monitors true 2024-04-25T07:40:59.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:00.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:00.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:00.514 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:41:00.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:41:00.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:00.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T07:41:01.132 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 98 up_thru 98 down_at 85 last_clean_interval [74,84) [v2:127.0.0.1:6826/1251301199,v1:127.0.0.1:6827/1251301199] [v2:127.0.0.1:6828/1251301199,v1:127.0.0.1:6829/1251301199] exists,up b4287a97-f3f4-4057-a2f6-cd858e9a523b 2024-04-25T07:41:01.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:41:01.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:41:01.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:41:01.133 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: cat td/osd-bluefs-volume-ops/osd.3.pid 2024-04-25T07:41:01.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:332: TEST_bluestore: osd_pid3=140913 2024-04-25T07:41:01.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:335: TEST_bluestore: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:41:01.253 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T07:41:01.253 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:41:01.253 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_141862 2024-04-25T07:41:01.253 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:41:01.253 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:41:02.253 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 113 97 0.378868 0.378906 0.179594 0.143288 2024-04-25T07:41:03.253 INFO:tasks.workunit.client.0.smithi134.stdout: 2 16 212 196 0.382757 0.386719 0.227126 0.151806 2024-04-25T07:41:04.253 INFO:tasks.workunit.client.0.smithi134.stdout: 3 16 335 319 0.415311 0.480469 0.124855 0.148997 2024-04-25T07:41:05.253 INFO:tasks.workunit.client.0.smithi134.stdout: 4 16 411 395 0.385695 0.296875 0.224636 0.154923 2024-04-25T07:41:06.253 INFO:tasks.workunit.client.0.smithi134.stdout: 5 16 515 499 0.389797 0.40625 0.179747 0.158039 2024-04-25T07:41:07.253 INFO:tasks.workunit.client.0.smithi134.stdout: 6 16 603 587 0.382116 0.34375 0.167799 0.160955 2024-04-25T07:41:08.254 INFO:tasks.workunit.client.0.smithi134.stdout: 7 16 723 707 0.394481 0.46875 0.149746 0.157219 2024-04-25T07:41:09.254 INFO:tasks.workunit.client.0.smithi134.stdout: 8 16 819 803 0.392039 0.375 0.208719 0.157363 2024-04-25T07:41:10.254 INFO:tasks.workunit.client.0.smithi134.stdout: 9 16 931 915 0.397083 0.4375 0.141868 0.155252 2024-04-25T07:41:11.254 INFO:tasks.workunit.client.0.smithi134.stdout: 10 16 1015 999 0.390184 0.328125 0.216896 0.159025 2024-04-25T07:41:12.254 INFO:tasks.workunit.client.0.smithi134.stdout: 11 16 1132 1116 0.396257 0.457031 0.116665 0.156516 2024-04-25T07:41:13.254 INFO:tasks.workunit.client.0.smithi134.stdout: 12 16 1237 1221 0.397409 0.410156 0.186738 0.156489 2024-04-25T07:41:14.254 INFO:tasks.workunit.client.0.smithi134.stdout: 13 16 1349 1333 0.400487 0.4375 0.165029 0.154961 2024-04-25T07:41:15.254 INFO:tasks.workunit.client.0.smithi134.stdout: 14 16 1453 1437 0.400896 0.40625 0.21746 0.155457 2024-04-25T07:41:16.255 INFO:tasks.workunit.client.0.smithi134.stdout: 15 16 1542 1526 0.397344 0.347656 0.132943 0.156897 2024-04-25T07:41:17.255 INFO:tasks.workunit.client.0.smithi134.stdout: 16 16 1630 1614 0.393991 0.34375 0.196087 0.157888 2024-04-25T07:41:18.255 INFO:tasks.workunit.client.0.smithi134.stdout: 17 16 1758 1742 0.400224 0.5 0.117417 0.155838 2024-04-25T07:41:19.258 INFO:tasks.workunit.client.0.smithi134.stdout: 18 16 1861 1845 0.40034 0.402344 0.185398 0.155423 2024-04-25T07:41:20.255 INFO:tasks.workunit.client.0.smithi134.stdout: 19 16 1979 1963 0.403526 0.460938 0.125475 0.154233 2024-04-25T07:41:21.255 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:41:21.260446+0000 min lat: 0.0648907 max lat: 0.392505 avg lat: 0.155316 2024-04-25T07:41:21.255 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:41:21.256 INFO:tasks.workunit.client.0.smithi134.stdout: 20 16 2059 2043 0.398972 0.3125 0.365788 0.155316 2024-04-25T07:41:22.255 INFO:tasks.workunit.client.0.smithi134.stdout: 21 16 2171 2155 0.400803 0.4375 0.133468 0.155248 2024-04-25T07:41:23.256 INFO:tasks.workunit.client.0.smithi134.stdout: 22 16 2277 2261 0.401403 0.414062 0.125081 0.155111 2024-04-25T07:41:24.256 INFO:tasks.workunit.client.0.smithi134.stdout: 23 16 2405 2389 0.405687 0.5 0.128901 0.153399 2024-04-25T07:41:25.256 INFO:tasks.workunit.client.0.smithi134.stdout: 24 16 2506 2490 0.40522 0.394531 0.149991 0.153764 2024-04-25T07:41:26.256 INFO:tasks.workunit.client.0.smithi134.stdout: 25 16 2630 2614 0.408383 0.484375 0.113099 0.152919 2024-04-25T07:41:27.256 INFO:tasks.workunit.client.0.smithi134.stdout: 26 16 2697 2681 0.402741 0.261719 0.186593 0.15485 2024-04-25T07:41:28.256 INFO:tasks.workunit.client.0.smithi134.stdout: 27 16 2841 2825 0.408655 0.5625 0.137752 0.152705 2024-04-25T07:41:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: 28 16 2939 2923 0.40773 0.382812 0.214476 0.152555 2024-04-25T07:41:30.257 INFO:tasks.workunit.client.0.smithi134.stdout: 29 16 3079 3063 0.412526 0.546875 0.113149 0.151358 2024-04-25T07:41:31.257 INFO:tasks.workunit.client.0.smithi134.stdout: 30 16 3189 3173 0.413096 0.429688 0.104825 0.15102 2024-04-25T07:41:32.263 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 30.1003 2024-04-25T07:41:32.263 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 3189 2024-04-25T07:41:32.263 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 4096 2024-04-25T07:41:32.263 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 4096 2024-04-25T07:41:32.263 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 0.413851 2024-04-25T07:41:32.263 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 0.0711433 2024-04-25T07:41:32.263 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 0.5625 2024-04-25T07:41:32.264 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 0.261719 2024-04-25T07:41:32.264 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 105 2024-04-25T07:41:32.264 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 18.2127 2024-04-25T07:41:32.264 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 144 2024-04-25T07:41:32.264 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 67 2024-04-25T07:41:32.264 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 0.150943 2024-04-25T07:41:32.264 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.0441756 2024-04-25T07:41:32.264 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 0.451567 2024-04-25T07:41:32.264 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.0648907 2024-04-25T07:41:32.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:337: TEST_bluestore: wait_for_clean 2024-04-25T07:41:32.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:41:32.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:41:32.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:41:32.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:41:32.351 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:41:32.351 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:41:32.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:41:32.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:41:32.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:41:32.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:41:32.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:41:32.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:41:32.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:41:32.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:41:32.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:41:32.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:41:32.886 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:41:32.886 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T07:41:32.886 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T07:41:32.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:41:32.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:41:32.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:41:33.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=373662154765 2024-04-25T07:41:33.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 373662154765 2024-04-25T07:41:33.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-373662154765' 2024-04-25T07:41:33.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:41:33.059 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:41:33.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=386547056652 2024-04-25T07:41:33.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 386547056652 2024-04-25T07:41:33.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-373662154765 1-386547056652' 2024-04-25T07:41:33.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:41:33.215 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:41:33.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=403726925835 2024-04-25T07:41:33.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 403726925835 2024-04-25T07:41:33.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-373662154765 1-386547056652 2-403726925835' 2024-04-25T07:41:33.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:41:33.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T07:41:33.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=420906795017 2024-04-25T07:41:33.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 420906795017 2024-04-25T07:41:33.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-373662154765 1-386547056652 2-403726925835 3-420906795017' 2024-04-25T07:41:33.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:41:33.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-373662154765 2024-04-25T07:41:33.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:41:33.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:41:33.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-373662154765 2024-04-25T07:41:33.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:41:33.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=373662154765 2024-04-25T07:41:33.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 373662154765' 2024-04-25T07:41:33.521 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 373662154765 2024-04-25T07:41:33.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:41:33.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 373662154763 -lt 373662154765 2024-04-25T07:41:33.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:41:34.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:41:34.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:41:35.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 373662154765 -lt 373662154765 2024-04-25T07:41:35.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:41:35.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:41:35.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-386547056652 2024-04-25T07:41:35.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:41:35.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-386547056652 2024-04-25T07:41:35.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:41:35.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=386547056652 2024-04-25T07:41:35.349 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 386547056652 2024-04-25T07:41:35.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 386547056652' 2024-04-25T07:41:35.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:41:35.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 386547056652 -lt 386547056652 2024-04-25T07:41:35.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:41:35.768 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-403726925835 2024-04-25T07:41:35.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:41:35.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:41:35.770 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-403726925835 2024-04-25T07:41:35.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:41:35.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=403726925835 2024-04-25T07:41:35.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 403726925835' 2024-04-25T07:41:35.772 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 403726925835 2024-04-25T07:41:35.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:41:36.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 403726925835 -lt 403726925835 2024-04-25T07:41:36.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:41:36.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:41:36.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-420906795017 2024-04-25T07:41:36.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T07:41:36.209 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-420906795017 2024-04-25T07:41:36.209 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:41:36.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=420906795017 2024-04-25T07:41:36.210 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 420906795017 2024-04-25T07:41:36.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 420906795017' 2024-04-25T07:41:36.211 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T07:41:36.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 420906795018 -lt 420906795017 2024-04-25T07:41:36.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:41:36.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:41:36.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:41:37.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:41:37.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:41:37.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:41:37.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:41:37.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:41:37.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:41:37.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:41:37.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:41:37.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:41:37.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:41:37.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:41:37.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:41:38.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:41:38.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:41:38.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:41:38.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:41:38.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:38.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:41:38.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:41:38.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:41:38.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:41:38.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:41:38.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:41:38.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:41:38.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:41:38.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:41:38.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:41:38.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:41:38.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:41:38.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:41:38.459 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:41:38.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:38.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:41:38.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:41:38.460 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:38.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:41:38.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:41:38.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:41:38.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:41:38.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:38.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:41:38.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95185 2024-04-25T07:41:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:41:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:41:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:41:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:41:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:41:38.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:38.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:41:38.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:41:38.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:41:38.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:41:38.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:41:38.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:41:38.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:41:38.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:41:38.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:41:38.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:41:38.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:41:38.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:38.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:41:38.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:41:38.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:41:38.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:41:38.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:41:38.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:41:38.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:41:38.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:38.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:41:38.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95185 2024-04-25T07:41:38.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:41:38.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:41:38.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:41:38.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:41:38.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:38.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:41:38.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.95185 2024-04-25T07:41:38.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:41:38.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:41:38.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:41:38.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:41:38.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore2 td/osd-bluefs-volume-ops 2024-04-25T07:41:38.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:341: TEST_bluestore2: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:38.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: ulimit -n 2024-04-25T07:41:38.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:343: TEST_bluestore2: local flimit=4096 2024-04-25T07:41:38.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:344: TEST_bluestore2: '[' 4096 -lt 1536 ']' 2024-04-25T07:41:38.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:41:38.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:347: TEST_bluestore2: CEPH_MON=127.0.0.1:7146 2024-04-25T07:41:38.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:348: TEST_bluestore2: export CEPH_ARGS 2024-04-25T07:41:38.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: uuidgen 2024-04-25T07:41:38.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:349: TEST_bluestore2: CEPH_ARGS+='--fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none ' 2024-04-25T07:41:38.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:350: TEST_bluestore2: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:351: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:352: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:353: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:354: TEST_bluestore2: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:355: TEST_bluestore2: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:356: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:357: TEST_bluestore2: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:358: TEST_bluestore2: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:360: TEST_bluestore2: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:41:38.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:41:38.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:41:38.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:39.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:41:39.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:41:39.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:41:39.434 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:41:39.434 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:39.434 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:41:39.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:41:39.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:41:39.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:41:39.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:41:39.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:41:39.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:41:39.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:41:39.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:41:39.655 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:41:39.656 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:41:39.656 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:41:39.657 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:41:39.657 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:39.657 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:41:39.657 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95185/ceph-mon.a.asok 2024-04-25T07:41:39.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:41:39.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95185/ceph-mon.a.asok config get fsid 2024-04-25T07:41:39.768 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:41:39.768 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:41:39.768 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:41:39.768 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:41:39.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:41:39.769 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:41:39.769 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:41:39.769 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:41:39.770 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:41:39.770 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:39.770 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:41:39.770 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95185/ceph-mon.a.asok 2024-04-25T07:41:39.770 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:41:39.770 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95185/ceph-mon.a.asok config get mon_host 2024-04-25T07:41:39.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:361: TEST_bluestore2: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:41:39.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:39.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:41:39.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:41:39.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:41:39.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:41:39.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:41:40.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:41:40.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:41:40.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:41:40.784 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:41:40.784 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:40.784 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:41:40.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:41:40.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:41:40.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:41:40.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:362: TEST_bluestore2: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:41:40.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:41:40.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:41:40.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:41:40.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:41:40.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:41:40.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:41:40.828 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:41:40.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:41:40.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4fdc89c9-5a52-499b-b497-5b401213d562 2024-04-25T07:41:40.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4fdc89c9-5a52-499b-b497-5b401213d562' 2024-04-25T07:41:40.830 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 4fdc89c9-5a52-499b-b497-5b401213d562 2024-04-25T07:41:40.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:41:40.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0CSpmz+64MhAAUIi4tXgsQ35sYwFlg71mxA== 2024-04-25T07:41:40.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0CSpmz+64MhAAUIi4tXgsQ35sYwFlg71mxA=="}' 2024-04-25T07:41:40.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4fdc89c9-5a52-499b-b497-5b401213d562 -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:41:41.148 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:41:41.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:41:41.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0CSpmz+64MhAAUIi4tXgsQ35sYwFlg71mxA== --osd-uuid 4fdc89c9-5a52-499b-b497-5b401213d562 2024-04-25T07:41:41.196 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:41.195+0000 7fadbde537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:41.216 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:41.215+0000 7fadbde537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:41.218 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:41.219+0000 7fadbde537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:41.218 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:41.219+0000 7fadbde537c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:41:41.218 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:41.219+0000 7fadbde537c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:41:41.218 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:41.219+0000 7fadbde537c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:41:41.248 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:41.247+0000 7fadbde537c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:41:49.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:41:49.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:41:49.612 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:41:49.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:41:49.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:41:50.198 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:41:50.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:41:50.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:41:50.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:41:50.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:41:50.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:41:50.235 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:50.235+0000 7f52836887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:50.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:50.255+0000 7f52836887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:50.256 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:50.255+0000 7f52836887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:41:50.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:41:50.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:41:50.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:41:50.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:41:50.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:41:50.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:50.630 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:41:50.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:41:50.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:50.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:51.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:51.080 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:51.079+0000 7f52836887c0 -1 Falling back to public interface 2024-04-25T07:41:52.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:52.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:52.036 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:41:52.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:41:52.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:52.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:52.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:53.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:53.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:53.447 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:41:53.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:41:53.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:53.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:53.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:54.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:54.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:54.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:41:54.931 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:41:54.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:54.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:55.175 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:41:55.175+0000 7f52836887c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:41:55.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:56.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:56.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:56.447 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:41:56.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:41:56.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:56.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:57.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:41:58.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:41:58.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:41:58.413 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:41:58.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:41:58.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:41:58.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:41:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:00.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:00.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:00.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:42:00.517 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:42:00.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:00.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:00.951 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2285938746,v1:127.0.0.1:6803/2285938746] [v2:127.0.0.1:6804/2285938746,v1:127.0.0.1:6805/2285938746] exists,up 4fdc89c9-5a52-499b-b497-5b401213d562 2024-04-25T07:42:00.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:42:00.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:42:00.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:42:00.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:42:00.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:363: TEST_bluestore2: osd_pid0=143758 2024-04-25T07:42:00.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:365: TEST_bluestore2: sleep 5 2024-04-25T07:42:05.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:366: TEST_bluestore2: create_pool foo 16 2024-04-25T07:42:05.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:42:06.499 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T07:42:06.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:42:07.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:368: TEST_bluestore2: retry = 0 2024-04-25T07:42:07.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 368: retry: command not found 2024-04-25T07:42:07.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:369: TEST_bluestore2: [[ '' -le 5 ]] 2024-04-25T07:42:07.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:371: TEST_bluestore2: timeout 60 rados bench -p foo 10 write --write-omap --no-cleanup 2024-04-25T07:42:07.624 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T07:42:07.626 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 4194304 bytes to objects of size 4194304 for up to 10 seconds or 0 objects 2024-04-25T07:42:07.626 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_145039 2024-04-25T07:42:07.661 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:42:07.662 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:42:08.662 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 17 1 3.99941 4 0.304746 0.304746 2024-04-25T07:42:09.662 INFO:tasks.workunit.client.0.smithi134.stdout: 2 16 33 17 33.9947 64 0.918388 1.00257 2024-04-25T07:42:10.662 INFO:tasks.workunit.client.0.smithi134.stdout: 3 16 49 33 43.9932 64 0.910054 0.986202 2024-04-25T07:42:11.662 INFO:tasks.workunit.client.0.smithi134.stdout: 4 16 65 49 48.9922 64 1.01903 1.00122 2024-04-25T07:42:12.662 INFO:tasks.workunit.client.0.smithi134.stdout: 5 16 81 65 51.9914 64 1.21122 1.0375 2024-04-25T07:42:13.673 INFO:tasks.workunit.client.0.smithi134.stdout: 6 16 97 81 53.9912 64 1.44537 1.11148 2024-04-25T07:42:14.663 INFO:tasks.workunit.client.0.smithi134.stdout: 7 16 112 96 54.8486 60 1.33386 1.14222 2024-04-25T07:42:15.663 INFO:tasks.workunit.client.0.smithi134.stdout: 8 16 129 113 56.4912 68 0.901964 1.09594 2024-04-25T07:42:16.663 INFO:tasks.workunit.client.0.smithi134.stdout: 9 16 144 128 56.8798 60 1.2165 1.10764 2024-04-25T07:42:17.663 INFO:tasks.workunit.client.0.smithi134.stdout: 10 16 160 144 57.5909 64 0.960357 1.09266 2024-04-25T07:42:18.663 INFO:tasks.workunit.client.0.smithi134.stdout: 11 15 160 145 52.719 4 1.02717 1.09221 2024-04-25T07:42:19.663 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 11.4132 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 160 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 4194304 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 4194304 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 56.0756 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 24.1871 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 68 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 4 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 14 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 6.04679 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 17 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 1 2024-04-25T07:42:19.664 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 1.1328 2024-04-25T07:42:19.665 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.219158 2024-04-25T07:42:19.665 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 1.5376 2024-04-25T07:42:19.665 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.304746 2024-04-25T07:42:19.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:374: TEST_bluestore2: sleep 10 2024-04-25T07:42:29.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:42:29.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: jq .bluefs.db_used_bytes 2024-04-25T07:42:29.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:376: TEST_bluestore2: db_used=1061158912 2024-04-25T07:42:29.910 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: ceph tell osd.0 perf dump bluefs 2024-04-25T07:42:29.910 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: jq .bluefs.slow_used_bytes 2024-04-25T07:42:30.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:377: TEST_bluestore2: spilled_over=36962304 2024-04-25T07:42:30.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:378: TEST_bluestore2: (( retry+=1 )) 2024-04-25T07:42:30.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: test 36962304 -eq 0 2024-04-25T07:42:30.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:379: TEST_bluestore2: break 2024-04-25T07:42:30.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:381: TEST_bluestore2: test 36962304 -gt 0 2024-04-25T07:42:30.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 143758 2024-04-25T07:42:30.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: sleep 1 2024-04-25T07:42:31.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:383: TEST_bluestore2: kill 143758 2024-04-25T07:42:31.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 383: kill: (143758) - No such process 2024-04-25T07:42:31.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:384: TEST_bluestore2: ceph osd down 0 2024-04-25T07:42:31.512 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:42:31.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:386: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --devs-source td/osd-bluefs-volume-ops/0/block.db --dev-target td/osd-bluefs-volume-ops/0/block --command bluefs-bdev-migrate 2024-04-25T07:42:31.555 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:42:41.097 INFO:tasks.workunit.client.0.smithi134.stdout: device removed:1 td/osd-bluefs-volume-ops/0/block.db 2024-04-25T07:42:41.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:391: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command bluefs-bdev-sizes 2024-04-25T07:42:41.145 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:42:42.934 INFO:tasks.workunit.client.0.smithi134.stdout:1 : device size 0xffffe000 : using 0x41957000(1.0 GiB) 2024-04-25T07:42:42.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:394: TEST_bluestore2: ceph-bluestore-tool --path td/osd-bluefs-volume-ops/0 --command fsck 2024-04-25T07:42:45.005 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:397: TEST_bluestore2: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:42:45.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:42:45.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:42:45.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:42:45.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:42:45.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:42:45.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:42:45.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:42:45.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:42:45.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:42:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:42:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:42:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:42:45.058 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:42:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:42:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:42:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:42:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:42:45.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:42:45.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:42:45.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:42:45.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:42:45.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:42:45.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:42:45.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:42:45.059 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:42:45.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:42:45.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:42:45.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:42:45.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:42:45.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:42:45.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:42:45.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:42:45.094 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:42:45.094+0000 7fd89c99e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:45.097 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:42:45.098+0000 7fd89c99e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:45.098 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:42:45.098+0000 7fd89c99e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:42:45.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:42:45.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:42:45.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:42:45.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:42:45.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:42:45.487 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:42:45.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:45.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:42:45.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:45.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:45.646 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:42:45.646+0000 7fd89c99e7c0 -1 Falling back to public interface 2024-04-25T07:42:45.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:46.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:46.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:46.936 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:42:46.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:42:46.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:47.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:48.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:48.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:48.348 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:42:48.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:42:48.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:48.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:48.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:49.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:49.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:49.781 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:42:49.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:42:49.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:49.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:50.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:51.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:51.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:51.208 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:42:51.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:42:51.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:51.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:51.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:52.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:52.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:52.622 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:42:52.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:42:52.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:52.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:53.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:53.437 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:42:53.438+0000 7fd89c99e7c0 -1 osd.0 10 log_to_monitors true 2024-04-25T07:42:54.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:54.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:54.034 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:42:54.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:42:54.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:54.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:54.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:55.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:55.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:55.918 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:42:55.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:42:55.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:55.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:57.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:42:58.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:42:58.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:42:58.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:42:58.771 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:42:58.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:42:58.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:42:59.304 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/961662740,v1:127.0.0.1:6803/961662740] [v2:127.0.0.1:6804/961662740,v1:127.0.0.1:6805/961662740] exists,up 4fdc89c9-5a52-499b-b497-5b401213d562 2024-04-25T07:42:59.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:42:59.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:42:59.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:42:59.305 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:42:59.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:398: TEST_bluestore2: osd_pid0=146264 2024-04-25T07:42:59.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:400: TEST_bluestore2: wait_for_clean 2024-04-25T07:42:59.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:42:59.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:42:59.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:42:59.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:42:59.307 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:42:59.307 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:42:59.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:42:59.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:42:59.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:42:59.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:42:59.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:42:59.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:42:59.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:42:59.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:42:59.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:42:59.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T07:42:59.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:42:59.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:42:59.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:43:00.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574852 2024-04-25T07:43:00.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574852 2024-04-25T07:43:00.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-55834574852' 2024-04-25T07:43:00.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:43:00.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-55834574852 2024-04-25T07:43:00.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:43:00.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:43:00.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574852 2024-04-25T07:43:00.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:43:00.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574852 2024-04-25T07:43:00.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-04-25T07:43:00.009 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 55834574852 2024-04-25T07:43:00.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:43:00.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 2 -lt 55834574852 2024-04-25T07:43:00.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:43:01.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:43:01.407 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:43:01.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574852 -lt 55834574852 2024-04-25T07:43:01.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:43:01.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:43:01.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:43:02.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:43:02.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:43:02.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:43:02.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:43:02.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:43:02.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:43:02.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:43:02.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:43:02.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:43:02.759 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:43:02.759 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:43:02.759 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:43:03.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:43:03.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:43:03.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:43:03.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:43:03.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:43:03.396 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:43:03.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:43:03.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:43:03.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:43:03.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:03.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:43:03.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:43:03.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:03.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:43:03.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:43:03.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:43:03.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95185 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:12: run: for func in $funcs 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:13: run: setup td/osd-bluefs-volume-ops 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-bluefs-volume-ops 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:43:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:43:03.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:43:03.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:43:03.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:43:03.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:43:03.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:43:03.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:43:03.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:43:03.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:43:03.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:43:03.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:43:03.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:43:03.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:43:03.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:03.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:43:03.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:43:03.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:43:03.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:43:03.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:43:03.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:43:03.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:43:03.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:03.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:43:03.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95185 2024-04-25T07:43:03.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:43:03.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:43:03.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-bluefs-volume-ops 2024-04-25T07:43:03.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:43:03.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:03.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:43:03.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.95185 2024-04-25T07:43:03.523 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:43:03.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:43:03.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:43:03.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-bluefs-volume-ops 1' TERM HUP INT 2024-04-25T07:43:03.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:14: run: TEST_bluestore_expand td/osd-bluefs-volume-ops 2024-04-25T07:43:03.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:404: TEST_bluestore_expand: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:43:03.524 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: ulimit -n 2024-04-25T07:43:03.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:406: TEST_bluestore_expand: local flimit=4096 2024-04-25T07:43:03.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:407: TEST_bluestore_expand: '[' 4096 -lt 1536 ']' 2024-04-25T07:43:03.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:43:03.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:410: TEST_bluestore_expand: CEPH_MON=127.0.0.1:7146 2024-04-25T07:43:03.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:411: TEST_bluestore_expand: export CEPH_ARGS 2024-04-25T07:43:03.524 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: uuidgen 2024-04-25T07:43:03.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:412: TEST_bluestore_expand: CEPH_ARGS+='--fsid=bda936f6-b678-48a3-9f2e-cc4f6663364a --auth-supported=none ' 2024-04-25T07:43:03.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:413: TEST_bluestore_expand: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:43:03.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:414: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_size=4294967296 ' 2024-04-25T07:43:03.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:415: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_create=true ' 2024-04-25T07:43:03.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:416: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_db_size=1073741824 ' 2024-04-25T07:43:03.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:417: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_block_wal_create=false ' 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:418: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_fsck_on_mount=true ' 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:419: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_size=1 ' 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:420: TEST_bluestore_expand: CEPH_ARGS+='--osd_pool_default_min_size=1 ' 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:421: TEST_bluestore_expand: CEPH_ARGS+='--bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:423: TEST_bluestore_expand: run_mon td/osd-bluefs-volume-ops a 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-bluefs-volume-ops/a 2024-04-25T07:43:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-bluefs-volume-ops/a --run-dir=td/osd-bluefs-volume-ops 2024-04-25T07:43:03.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:43:03.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:03.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:03.986 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:03.986 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:03.986 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:43:03.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:43:03.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-bluefs-volume-ops/a '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --mon-cluster-log-file=td/osd-bluefs-volume-ops/log --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:43:04.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:43:04.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:43:04.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:43:04.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:43:04.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:43:04.232 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:43:04.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:43:04.232 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:43:04.232 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:43:04.232 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:43:04.232 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:04.233 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:43:04.233 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95185/ceph-mon.a.asok 2024-04-25T07:43:04.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:43:04.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95185/ceph-mon.a.asok config get fsid 2024-04-25T07:43:04.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:43:04.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:43:04.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:43:04.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:43:04.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:43:04.350 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:43:04.350 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:43:04.350 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:43:04.350 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:43:04.350 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:04.350 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:43:04.351 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.95185/ceph-mon.a.asok 2024-04-25T07:43:04.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:43:04.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.95185/ceph-mon.a.asok config get mon_host 2024-04-25T07:43:04.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:424: TEST_bluestore_expand: run_mgr td/osd-bluefs-volume-ops x 2024-04-25T07:43:04.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:43:04.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:43:04.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:43:04.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:43:04.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-bluefs-volume-ops/x 2024-04-25T07:43:04.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:43:05.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:43:05.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:05.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:05.369 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:05.369 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:05.369 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:43:05.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:43:05.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:43:05.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-bluefs-volume-ops/x '--log-file=td/osd-bluefs-volume-ops/$name.log' '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --run-dir=td/osd-bluefs-volume-ops '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:43:05.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:425: TEST_bluestore_expand: run_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:43:05.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:43:05.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:43:05.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:43:05.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:43:05.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:43:05.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=bda936f6-b678-48a3-9f2e-cc4f6663364a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:43:05.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:43:05.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:43:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:43:05.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:43:05.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:43:05.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:43:05.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:43:05.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=25dcc839-7e38-4a6f-ba5a-9ecd725acbd5 2024-04-25T07:43:05.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 25dcc839-7e38-4a6f-ba5a-9ecd725acbd5' 2024-04-25T07:43:05.450 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 25dcc839-7e38-4a6f-ba5a-9ecd725acbd5 2024-04-25T07:43:05.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:43:05.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCJCSpm9vaHHBAA34sqYWoysw95lPMwAEKbVQ== 2024-04-25T07:43:05.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCJCSpm9vaHHBAA34sqYWoysw95lPMwAEKbVQ=="}' 2024-04-25T07:43:05.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 25dcc839-7e38-4a6f-ba5a-9ecd725acbd5 -i td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:43:05.808 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:43:05.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-bluefs-volume-ops/0/new.json 2024-04-25T07:43:05.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=bda936f6-b678-48a3-9f2e-cc4f6663364a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCJCSpm9vaHHBAA34sqYWoysw95lPMwAEKbVQ== --osd-uuid 25dcc839-7e38-4a6f-ba5a-9ecd725acbd5 2024-04-25T07:43:05.871 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:05.870+0000 7f87b02ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:05.885 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:05.886+0000 7f87b02ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:05.886 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:05.886+0000 7f87b02ce7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:05.887 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:05.886+0000 7f87b02ce7c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:43:05.887 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:05.886+0000 7f87b02ce7c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:43:05.887 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:05.886+0000 7f87b02ce7c0 -1 bluestore(td/osd-bluefs-volume-ops/0/block) _read_bdev_label failed to open td/osd-bluefs-volume-ops/0/block: (2) No such file or directory 2024-04-25T07:43:05.925 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:05.926+0000 7f87b02ce7c0 -1 bluestore(td/osd-bluefs-volume-ops/0) _read_fsid unparsable uuid 2024-04-25T07:43:14.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-bluefs-volume-ops/0/keyring 2024-04-25T07:43:14.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:43:14.271 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:43:14.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:43:14.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-bluefs-volume-ops/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:43:14.849 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:43:14.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:43:14.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:43:14.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=bda936f6-b678-48a3-9f2e-cc4f6663364a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:43:14.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:43:14.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:43:14.893 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:14.898+0000 7fcce55767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:14.922 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:14.926+0000 7fcce55767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:14.924 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:14.926+0000 7fcce55767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:43:15.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:43:15.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:43:15.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:43:15.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:43:15.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:43:15.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:15.293 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:43:15.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:43:15.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:15.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:15.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:15.742 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:15.745+0000 7fcce55767c0 -1 Falling back to public interface 2024-04-25T07:43:16.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:16.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:16.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:43:16.711 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:43:16.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:16.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:17.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:18.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:18.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:18.154 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:43:18.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:43:18.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:18.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:19.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:19.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:19.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:43:19.567 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:43:19.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:19.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:19.872 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:43:19.873+0000 7fcce55767c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:43:20.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:21.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:21.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:21.013 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:43:21.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:43:21.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:21.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:23.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:23.602 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:43:23.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:23.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:43:23.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:23.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:24.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:43:25.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:43:25.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:43:25.322 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:43:25.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:43:25.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:43:25.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:43:25.776 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2188232126,v1:127.0.0.1:6803/2188232126] [v2:127.0.0.1:6804/2188232126,v1:127.0.0.1:6805/2188232126] exists,up 25dcc839-7e38-4a6f-ba5a-9ecd725acbd5 2024-04-25T07:43:25.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:43:25.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:43:25.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:43:25.777 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:43:25.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:426: TEST_bluestore_expand: osd_pid0=149007 2024-04-25T07:43:25.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:428: TEST_bluestore_expand: sleep 5 2024-04-25T07:43:30.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:429: TEST_bluestore_expand: create_pool foo 16 2024-04-25T07:43:30.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-04-25T07:43:31.309 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T07:43:31.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:43:32.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:432: TEST_bluestore_expand: timeout 60 rados bench -p foo 30 write -b 4096 --no-cleanup 2024-04-25T07:43:32.469 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T07:43:32.469 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2024-04-25T07:43:32.469 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_150033 2024-04-25T07:43:32.469 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:43:32.470 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:43:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 97 81 0.316359 0.316406 0.116893 0.175273 2024-04-25T07:43:34.470 INFO:tasks.workunit.client.0.smithi134.stdout: 2 16 241 225 0.439389 0.5625 0.108704 0.138706 2024-04-25T07:43:35.470 INFO:tasks.workunit.client.0.smithi134.stdout: 3 16 353 337 0.438738 0.4375 0.117063 0.138293 2024-04-25T07:43:36.470 INFO:tasks.workunit.client.0.smithi134.stdout: 4 16 481 465 0.454031 0.5 0.117101 0.13506 2024-04-25T07:43:37.470 INFO:tasks.workunit.client.0.smithi134.stdout: 5 16 609 593 0.463214 0.5 0.108745 0.132982 2024-04-25T07:43:38.471 INFO:tasks.workunit.client.0.smithi134.stdout: 6 16 737 721 0.469332 0.5 0.10034 0.131091 2024-04-25T07:43:39.470 INFO:tasks.workunit.client.0.smithi134.stdout: 7 16 865 849 0.473703 0.5 0.158879 0.129682 2024-04-25T07:43:40.471 INFO:tasks.workunit.client.0.smithi134.stdout: 8 16 993 977 0.476982 0.5 0.10043 0.130174 2024-04-25T07:43:41.471 INFO:tasks.workunit.client.0.smithi134.stdout: 9 16 1105 1089 0.47259 0.4375 0.125431 0.130681 2024-04-25T07:43:42.471 INFO:tasks.workunit.client.0.smithi134.stdout: 10 16 1232 1216 0.474933 0.496094 0.125778 0.13135 2024-04-25T07:43:43.471 INFO:tasks.workunit.client.0.smithi134.stdout: 11 16 1360 1344 0.477206 0.5 0.133631 0.130805 2024-04-25T07:43:44.471 INFO:tasks.workunit.client.0.smithi134.stdout: 12 16 1473 1457 0.474217 0.441406 0.100454 0.130739 2024-04-25T07:43:45.471 INFO:tasks.workunit.client.0.smithi134.stdout: 13 16 1601 1585 0.476196 0.5 0.117084 0.130414 2024-04-25T07:43:46.472 INFO:tasks.workunit.client.0.smithi134.stdout: 14 16 1728 1712 0.477613 0.496094 0.116658 0.130591 2024-04-25T07:43:47.472 INFO:tasks.workunit.client.0.smithi134.stdout: 15 16 1841 1825 0.475194 0.441406 0.0920645 0.131148 2024-04-25T07:43:48.472 INFO:tasks.workunit.client.0.smithi134.stdout: 16 16 1969 1953 0.476739 0.5 0.117031 0.130238 2024-04-25T07:43:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: 17 16 2097 2081 0.478104 0.5 0.133821 0.130214 2024-04-25T07:43:50.472 INFO:tasks.workunit.client.0.smithi134.stdout: 18 16 2225 2209 0.479316 0.5 0.10871 0.129626 2024-04-25T07:43:51.472 INFO:tasks.workunit.client.0.smithi134.stdout: 19 16 2368 2352 0.483484 0.558594 0.100251 0.129102 2024-04-25T07:43:52.472 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:43:52.477805+0000 min lat: 0.0906034 max lat: 0.24349 avg lat: 0.128527 2024-04-25T07:43:52.473 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:43:52.473 INFO:tasks.workunit.client.0.smithi134.stdout: 20 16 2497 2481 0.484503 0.503906 0.1087 0.128527 2024-04-25T07:43:53.473 INFO:tasks.workunit.client.0.smithi134.stdout: 21 16 2609 2593 0.482261 0.4375 0.117082 0.129018 2024-04-25T07:43:54.473 INFO:tasks.workunit.client.0.smithi134.stdout: 22 16 2751 2735 0.48555 0.554688 0.109156 0.128647 2024-04-25T07:43:55.473 INFO:tasks.workunit.client.0.smithi134.stdout: 23 16 2833 2817 0.478362 0.320312 0.125679 0.129261 2024-04-25T07:43:56.473 INFO:tasks.workunit.client.0.smithi134.stdout: 24 16 2928 2912 0.473891 0.371094 0.118094 0.131841 2024-04-25T07:43:57.473 INFO:tasks.workunit.client.0.smithi134.stdout: 25 16 3041 3025 0.47259 0.441406 0.133855 0.131654 2024-04-25T07:43:58.473 INFO:tasks.workunit.client.0.smithi134.stdout: 26 16 3184 3168 0.475896 0.558594 0.109618 0.131122 2024-04-25T07:43:59.473 INFO:tasks.workunit.client.0.smithi134.stdout: 27 16 3312 3296 0.476785 0.5 0.109172 0.130983 2024-04-25T07:44:00.474 INFO:tasks.workunit.client.0.smithi134.stdout: 28 16 3425 3409 0.475519 0.441406 0.133828 0.130997 2024-04-25T07:44:01.474 INFO:tasks.workunit.client.0.smithi134.stdout: 29 16 3553 3537 0.476361 0.5 0.116976 0.130982 2024-04-25T07:44:02.474 INFO:tasks.workunit.client.0.smithi134.stdout: 30 16 3665 3649 0.475063 0.4375 0.117049 0.130998 2024-04-25T07:44:03.474 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 30.0637 2024-04-25T07:44:03.474 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 3665 2024-04-25T07:44:03.474 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 4096 2024-04-25T07:44:03.474 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 4096 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 0.476203 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 0.0608058 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 0.5625 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 0.316406 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 121 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 15.5663 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 144 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 81 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 0.131062 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.0375603 2024-04-25T07:44:03.475 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 0.63411 2024-04-25T07:44:03.476 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.0906034 2024-04-25T07:44:03.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:433: TEST_bluestore_expand: sleep 5 2024-04-25T07:44:08.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-04-25T07:44:08.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-04-25T07:44:08.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:435: TEST_bluestore_expand: total_space_before=4294967296 2024-04-25T07:44:08.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-04-25T07:44:08.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-04-25T07:44:08.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: grep free 2024-04-25T07:44:08.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d: -f 2 2024-04-25T07:44:08.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut -d, -f 1 2024-04-25T07:44:08.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: cut '-d ' -f 2 2024-04-25T07:44:08.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:436: TEST_bluestore_expand: free_space_before=4279853056 2024-04-25T07:44:08.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 149007 2024-04-25T07:44:08.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: sleep 1 2024-04-25T07:44:09.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:439: TEST_bluestore_expand: kill 149007 2024-04-25T07:44:09.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 439: kill: (149007) - No such process 2024-04-25T07:44:09.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:440: TEST_bluestore_expand: ceph osd down 0 2024-04-25T07:44:10.340 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:44:10.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:443: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 allocmap 2024-04-25T07:44:10.389 INFO:tasks.workunit.client.0.smithi134.stdout:allocmap bluestore.allocmap 2024-04-25T07:44:13.173 INFO:tasks.workunit.client.0.smithi134.stdout:allocmap success 2024-04-25T07:44:13.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:446: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:44:16.257 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:44:16.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:448: TEST_bluestore_expand: requested_space=4294967296 2024-04-25T07:44:16.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:449: TEST_bluestore_expand: truncate td/osd-bluefs-volume-ops/0/block -s 4294967296 2024-04-25T07:44:16.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:450: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-expand 2024-04-25T07:44:16.315 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:44:18.123 INFO:tasks.workunit.client.0.smithi134.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-04-25T07:44:18.123 INFO:tasks.workunit.client.0.smithi134.stdout:2 : device size 0x100000000 : using 0xe6a000(14 MiB) 2024-04-25T07:44:18.123 INFO:tasks.workunit.client.0.smithi134.stdout:Expanding DB/WAL... 2024-04-25T07:44:18.123 INFO:tasks.workunit.client.0.smithi134.stdout:1 : expanding to 0x1073741824 2024-04-25T07:44:18.150 INFO:tasks.workunit.client.0.smithi134.stdout:1 : size label updated to 1073741824 2024-04-25T07:44:18.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:453: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 fsck 2024-04-25T07:44:21.685 INFO:tasks.workunit.client.0.smithi134.stdout:fsck success 2024-04-25T07:44:21.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:456: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-04-25T07:44:21.743 INFO:tasks.workunit.client.0.smithi134.stdout:qfsck bluestore.quick-fsck 2024-04-25T07:44:24.529 INFO:tasks.workunit.client.0.smithi134.stdout:qfsck success 2024-04-25T07:44:24.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:458: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 bluefs-bdev-sizes 2024-04-25T07:44:24.576 INFO:tasks.workunit.client.0.smithi134.stdout:inferring bluefs devices from bluestore path 2024-04-25T07:44:27.359 INFO:tasks.workunit.client.0.smithi134.stdout:1 : device size 0x3fffe000 : using 0x2300000(35 MiB) 2024-04-25T07:44:27.359 INFO:tasks.workunit.client.0.smithi134.stdout:2 : device size 0x100000000 : using 0xe6a000(14 MiB) 2024-04-25T07:44:27.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:460: TEST_bluestore_expand: activate_osd td/osd-bluefs-volume-ops 0 2024-04-25T07:44:27.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:44:27.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:44:27.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-bluefs-volume-ops/0 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=bda936f6-b678-48a3-9f2e-cc4f6663364a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd ' 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-bluefs-volume-ops/0' 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-bluefs-volume-ops/0/journal' 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-bluefs-volume-ops' 2024-04-25T07:44:27.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:44:27.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:27.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:27.370 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:27.370 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:27.370 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:44:27.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:44:27.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' 2024-04-25T07:44:27.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:44:27.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-bluefs-volume-ops/$name.log' 2024-04-25T07:44:27.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-bluefs-volume-ops/$name.pid' 2024-04-25T07:44:27.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:44:27.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:44:27.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:44:27.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:44:27.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:44:27.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:44:27.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-bluefs-volume-ops/0 2024-04-25T07:44:27.371 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:44:27.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:44:27.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=24a0cb97-72be-4598-98c3-e4bf83dc978c --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=2147483648 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_size=536870912 --bluestore_block_wal_create=true --bluestore_fsck_on_mount=true --fsid=21b9dc27-a142-4ebc-a5e2-a390c5d61ea6 --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --fsid=bda936f6-b678-48a3-9f2e-cc4f6663364a --auth-supported=none --mon-host=127.0.0.1:7146 --bluestore_block_size=4294967296 --bluestore_block_db_create=true --bluestore_block_db_size=1073741824 --bluestore_block_wal_create=false --bluestore_fsck_on_mount=true --osd_pool_default_size=1 --osd_pool_default_min_size=1 --bluestore_debug_enforce_settings=ssd --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-bluefs-volume-ops/0 --osd-journal=td/osd-bluefs-volume-ops/0/journal --chdir= --run-dir=td/osd-bluefs-volume-ops '--admin-socket=/tmp/ceph-asok.95185/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-bluefs-volume-ops/$name.log' '--pid-file=td/osd-bluefs-volume-ops/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:44:27.372 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-bluefs-volume-ops/0/whoami 2024-04-25T07:44:27.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:44:27.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:44:27.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:44:27.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:44:27.415 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:27.416+0000 7f16355687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:27.418 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:27.420+0000 7f16355687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:27.419 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:27.420+0000 7f16355687c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:27.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:44:27.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:44:27.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:44:27.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:44:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:44:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:27.992 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:44:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:44:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:28.203 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:28.204+0000 7f16355687c0 -1 Falling back to public interface 2024-04-25T07:44:29.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:30.213 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:44:30.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:30.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:30.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:44:30.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:30.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:30.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:31.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:31.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:31.663 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:44:31.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:44:31.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:31.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:32.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:33.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:33.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:33.093 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:44:33.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:44:33.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:33.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:33.286 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:33.287+0000 7f16355687c0 -1 osd.0 10 log_to_monitors true 2024-04-25T07:44:33.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:34.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:34.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:34.568 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:44:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:44:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:34.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:35.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:36.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:36.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:36.681 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:44:36.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:44:36.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:36.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:37.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:38.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:38.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:38.192 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:44:38.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:44:38.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:38.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:38.644 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 13 up_thru 13 down_at 11 last_clean_interval [5,10) [v2:127.0.0.1:6802/2566820496,v1:127.0.0.1:6803/2566820496] [v2:127.0.0.1:6804/2566820496,v1:127.0.0.1:6805/2566820496] exists,up 25dcc839-7e38-4a6f-ba5a-9ecd725acbd5 2024-04-25T07:44:38.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:44:38.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:44:38.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:44:38.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: cat td/osd-bluefs-volume-ops/osd.0.pid 2024-04-25T07:44:38.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:461: TEST_bluestore_expand: osd_pid0=152384 2024-04-25T07:44:38.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:463: TEST_bluestore_expand: wait_for_clean 2024-04-25T07:44:38.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:44:38.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:44:38.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:44:38.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:44:38.647 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:44:38.647 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:44:38.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:44:38.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:44:38.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:44:38.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:44:38.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:44:38.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:44:38.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:44:38.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:44:38.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:44:39.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T07:44:39.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:44:39.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:44:39.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:44:39.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574852 2024-04-25T07:44:39.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574852 2024-04-25T07:44:39.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-55834574852' 2024-04-25T07:44:39.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:44:39.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:44:39.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-55834574852 2024-04-25T07:44:39.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:44:39.337 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-55834574852 2024-04-25T07:44:39.337 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:44:39.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574852 2024-04-25T07:44:39.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 55834574852' 2024-04-25T07:44:39.338 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 55834574852 2024-04-25T07:44:39.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:44:39.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 55834574852 2024-04-25T07:44:39.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:44:40.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:44:40.753 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:44:41.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574852 -lt 55834574852 2024-04-25T07:44:41.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:44:41.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:44:41.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:44:41.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 16 == 0 2024-04-25T07:44:41.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:44:41.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:44:41.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:44:41.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:44:41.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:44:41.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:44:41.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:44:42.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=16 2024-04-25T07:44:42.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:44:42.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:44:42.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:44:42.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 16 = 16 2024-04-25T07:44:42.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:44:42.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:44:42.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: ceph tell osd.0 perf dump bluefs 2024-04-25T07:44:42.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: jq .bluefs.slow_total_bytes 2024-04-25T07:44:42.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:465: TEST_bluestore_expand: total_space_after=4294967296 2024-04-25T07:44:42.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: ceph tell osd.0 bluestore bluefs device info 2024-04-25T07:44:42.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep BDEV_SLOW -A 2 2024-04-25T07:44:42.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: grep free 2024-04-25T07:44:42.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d: -f 2 2024-04-25T07:44:42.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut -d, -f 1 2024-04-25T07:44:42.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: cut '-d ' -f 2 2024-04-25T07:44:43.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:466: TEST_bluestore_expand: free_space_after=4279812096 2024-04-25T07:44:43.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:468: TEST_bluestore_expand: '[4294967296' '!=' '4294967296]' 2024-04-25T07:44:43.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 468: [4294967296: command not found 2024-04-25T07:44:43.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:474: TEST_bluestore_expand: total_space_added=0 2024-04-25T07:44:43.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:475: TEST_bluestore_expand: free_space_added=-40960 2024-04-25T07:44:43.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:477: TEST_bluestore_expand: let 'new_used_space=(0 - -40960)' 2024-04-25T07:44:43.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:478: TEST_bluestore_expand: echo 40960 2024-04-25T07:44:43.113 INFO:tasks.workunit.client.0.smithi134.stdout:40960 2024-04-25T07:44:43.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:480: TEST_bluestore_expand: '[' 40960 -gt 131072 ']' 2024-04-25T07:44:43.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 152384 2024-04-25T07:44:43.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: sleep 1 2024-04-25T07:44:44.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:487: TEST_bluestore_expand: kill 152384 2024-04-25T07:44:44.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 487: kill: (152384) - No such process 2024-04-25T07:44:44.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:488: TEST_bluestore_expand: ceph osd down 0 2024-04-25T07:44:44.530 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T07:44:44.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:490: TEST_bluestore_expand: ceph-bluestore-tool --log-file td/osd-bluefs-volume-ops/bluestore_tool.log --path td/osd-bluefs-volume-ops/0 qfsck 2024-04-25T07:44:44.580 INFO:tasks.workunit.client.0.smithi134.stdout:qfsck bluestore.quick-fsck 2024-04-25T07:44:46.633 INFO:tasks.workunit.client.0.smithi134.stdout:qfsck success 2024-04-25T07:44:46.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh:15: run: teardown td/osd-bluefs-volume-ops 2024-04-25T07:44:46.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:44:46.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:46.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:44:46.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:46.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:46.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:46.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:46.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:46.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:46.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:46.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:46.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:46.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:46.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:46.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:46.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:46.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:46.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:44:46.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:46.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:46.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:44:46.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95185 2024-04-25T07:44:46.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:46.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:46.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:44:46.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-bluefs-volume-ops 0 2024-04-25T07:44:46.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-bluefs-volume-ops 2024-04-25T07:44:46.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:44:46.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-bluefs-volume-ops KILL 2024-04-25T07:44:46.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:46.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:46.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:46.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:46.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:46.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:46.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:46.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:46.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:46.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:46.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:46.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:46.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:46.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:46.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:46.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:44:46.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-bluefs-volume-ops 2024-04-25T07:44:46.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:46.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:46.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.95185 2024-04-25T07:44:46.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.95185 2024-04-25T07:44:46.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:46.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:46.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:44:46.850 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:44:46.850 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:44:46.861 INFO:tasks.workunit:Running workunit osd/osd-config.sh... 2024-04-25T07:44:46.862 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh 2024-04-25T07:44:46.915 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:44:46.920 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:44:46.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:44:46.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:44:46.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:44:46.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:44:46.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-config 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:22: run: local dir=td/osd-config 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:23: run: shift 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: export CEPH_MON=127.0.0.1:7100 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:25: run: CEPH_MON=127.0.0.1:7100 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:26: run: export CEPH_ARGS 2024-04-25T07:44:46.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: uuidgen 2024-04-25T07:44:46.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:27: run: CEPH_ARGS+='--fsid=00c37098-7115-49ca-8673-1548f53aa490 --auth-supported=none ' 2024-04-25T07:44:46.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-04-25T07:44:46.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: set 2024-04-25T07:44:46.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:30: run: local 'funcs=TEST_config_init 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_config_track 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_default_adjustment' 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:44:46.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:44:46.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:44:46.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:44:46.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:44:46.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:44:46.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:44:46.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:44:46.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:44:46.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:44:46.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:44:46.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:44:46.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:44:46.928 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:44:46.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:44:46.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:44:46.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:44:46.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:44:46.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:44:46.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:44:46.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:44:46.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:46.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:44:46.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154105 2024-04-25T07:44:46.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:44:46.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:44:46.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:44:46.934 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:44:46.934 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:46.934 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:44:46.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.154105 2024-04-25T07:44:46.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:44:46.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:44:46.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_init td/osd-config 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:39: TEST_config_init: local dir=td/osd-config 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:41: TEST_config_init: run_mon td/osd-config a 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-04-25T07:44:46.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-04-25T07:44:47.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:44:47.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:47.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:47.405 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:47.405 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:47.405 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:44:47.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154105/$cluster-$name.asok' 2024-04-25T07:44:47.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:44:47.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:44:47.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:44:47.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:47.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:47.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:44:47.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:44:47.645 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:47.646 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:47.646 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:47.646 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:47.646 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:47.646 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:44:47.646 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154105/ceph-mon.a.asok 2024-04-25T07:44:47.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:47.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154105/ceph-mon.a.asok config get fsid 2024-04-25T07:44:47.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:44:47.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:44:47.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:44:47.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:44:47.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:44:47.763 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:44:47.763 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:44:47.764 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:44:47.764 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:44:47.764 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:47.764 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:44:47.764 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154105/ceph-mon.a.asok 2024-04-25T07:44:47.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:44:47.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154105/ceph-mon.a.asok config get mon_host 2024-04-25T07:44:47.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:42: TEST_config_init: run_mgr td/osd-config x 2024-04-25T07:44:47.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-04-25T07:44:47.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:44:47.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:44:47.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:44:47.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-04-25T07:44:47.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:44:48.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:44:48.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:48.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:48.838 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:48.838 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:48.839 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:44:48.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154105/$cluster-$name.asok' 2024-04-25T07:44:48.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:48.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:44:48.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:43: TEST_config_init: local stale=1000 2024-04-25T07:44:48.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:44: TEST_config_init: local cache=500 2024-04-25T07:44:48.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:45: TEST_config_init: run_osd td/osd-config 0 --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=00c37098-7115-49ca-8673-1548f53aa490 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:44:48.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154105/$cluster-$name.asok' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-04-25T07:44:48.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:44:48.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:44:48.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:44:48.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:44:48.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:44:48.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000' 2024-04-25T07:44:48.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-04-25T07:44:48.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:44:48.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=00d05a9f-2ba2-4d46-982e-651e8dadd5f0 2024-04-25T07:44:48.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 00d05a9f-2ba2-4d46-982e-651e8dadd5f0' 2024-04-25T07:44:48.881 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 00d05a9f-2ba2-4d46-982e-651e8dadd5f0 2024-04-25T07:44:48.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:44:48.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDwCSpmNY+hNRAAhV0hcXW0oYEd1MmkiHUvzA== 2024-04-25T07:44:48.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDwCSpmNY+hNRAAhV0hcXW0oYEd1MmkiHUvzA=="}' 2024-04-25T07:44:48.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 00d05a9f-2ba2-4d46-982e-651e8dadd5f0 -i td/osd-config/0/new.json 2024-04-25T07:44:49.245 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:44:49.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-04-25T07:44:49.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=00c37098-7115-49ca-8673-1548f53aa490 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 --mkfs --key AQDwCSpmNY+hNRAAhV0hcXW0oYEd1MmkiHUvzA== --osd-uuid 00d05a9f-2ba2-4d46-982e-651e8dadd5f0 2024-04-25T07:44:49.297 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:49.299+0000 7f7c2d1ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:49.330 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:49.331+0000 7f7c2d1ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:49.331 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:49.331+0000 7f7c2d1ca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:49.332 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:49.331+0000 7f7c2d1ca7c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:44:49.332 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:49.331+0000 7f7c2d1ca7c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:44:49.332 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:49.331+0000 7f7c2d1ca7c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:44:49.370 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:49.371+0000 7f7c2d1ca7c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-04-25T07:44:57.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-04-25T07:44:57.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:44:57.385 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:44:57.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:44:57.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:44:57.964 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:44:57.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:44:57.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=00c37098-7115-49ca-8673-1548f53aa490 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-map-cache-size=500 --osd-pg-epoch-persisted-max-stale=1000 2024-04-25T07:44:57.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:44:57.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:44:57.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:44:57.995 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:57.995+0000 7f47c17e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:57.997 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:57.999+0000 7f47c17e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:57.999+0000 7f47c17e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:44:58.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:44:58.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:44:58.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:44:58.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:44:58.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:44:58.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:58.353 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:44:58.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:44:58.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:58.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:44:58.531 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:44:58.531+0000 7f47c17e57c0 -1 Falling back to public interface 2024-04-25T07:44:58.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:44:59.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:44:59.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:44:59.803 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:44:59.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:44:59.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:44:59.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:00.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:01.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:01.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:01.223 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:45:01.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:45:01.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:01.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:01.387 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:01.387+0000 7f47c17e57c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:45:01.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:02.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:02.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:02.645 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:45:02.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:45:02.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:02.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:03.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:04.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:04.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:04.093 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:45:04.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:45:04.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:04.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:04.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:05.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:05.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:05.640 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:45:05.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:45:05.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:06.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:07.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:07.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:07.084 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:45:07.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:45:07.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:07.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:07.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:08.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:08.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:08.565 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:45:08.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:45:08.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:08.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:09.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:10.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:10.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:10.038 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:45:10.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:45:10.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:10.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:10.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:11.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:11.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:11.576 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:45:11.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:45:11.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:11.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:12.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:13.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:13.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:13.044 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:45:13.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:45:13.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:13.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:13.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:14.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:14.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:14.607 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:45:14.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:45:14.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:14.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:15.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:16.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:16.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:16.100 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:45:16.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:45:16.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:16.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:16.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:17.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:17.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:17.834 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:45:17.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:45:17.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:17.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:18.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:19.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:19.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:19.427 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:45:19.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:45:19.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:19.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:19.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:20.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:20.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:20.870 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:45:20.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:45:20.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:20.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:21.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:22.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:22.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:22.457 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:45:22.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:45:22.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:22.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:23.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:24.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:24.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:24.245 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:45:24.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:45:24.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:24.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:25.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:26.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:26.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:26.012 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:45:26.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:45:26.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:26.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:27.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:28.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:28.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:28.399 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:45:28.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:45:28.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:28.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:29.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:30.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:30.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:30.309 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:45:30.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:45:30.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:30.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:31.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:32.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:32.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:32.310 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:45:32.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:45:32.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:32.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:32.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:33.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:33.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:33.973 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:45:33.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:45:33.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:33.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:34.391 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3406248150,v1:127.0.0.1:6803/3406248150] [v2:127.0.0.1:6804/3406248150,v1:127.0.0.1:6805/3406248150] exists,up 00d05a9f-2ba2-4d46-982e-651e8dadd5f0 2024-04-25T07:45:34.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:45:34.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:45:34.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:45:34.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: get_asok_path osd.0 2024-04-25T07:45:34.393 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:45:34.393 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:45:34.393 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:45:34.393 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:34.393 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:45:34.393 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154105/ceph-osd.0.asok 2024-04-25T07:45:34.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: CEPH_ARGS= 2024-04-25T07:45:34.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:49: TEST_config_init: ceph --admin-daemon /tmp/ceph-asok.154105/ceph-osd.0.asok log flush 2024-04-25T07:45:34.501 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T07:45:34.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:50: TEST_config_init: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:45:34.516 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:45:01.795+0000 7f47c17e57c0 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:45:34.516 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:45:04.387+0000 7f47b1787640 10 log_client will send 2024-04-25T07:45:01.798864+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:45:34.516 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:45:04.739+0000 7f47b2789640 10 log_client logged 2024-04-25T07:45:01.798864+0000 osd.0 (osd.0) 1 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:45:34.516 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:45:33.258+0000 7f47bd79f640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:45:34.516 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:45:33.394+0000 7f47b1787640 10 log_client will send 2024-04-25T07:45:33.262674+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:45:34.516 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:45:33.674+0000 7f47b2789640 10 log_client logged 2024-04-25T07:45:33.262674+0000 osd.0 (osd.0) 4 : cluster [WRN] osd_map_cache_size (500) is not > osd_pg_epoch_persisted_max_stale (1000) 2024-04-25T07:45:34.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:45:34.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:45:34.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:45:34.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:45:34.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:45:34.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:45:34.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:45:34.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:45:34.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:45:34.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:45:34.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:45:34.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:45:34.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:45:34.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:45:34.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:45:34.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:45:34.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:34.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:45:34.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:45:34.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:34.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:45:34.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:45:34.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:45:34.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:45:34.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:34.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:45:34.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154105 2024-04-25T07:45:34.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:45:34.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:45:34.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:45:34.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:45:34.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:45:34.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:45:34.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:45:34.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:45:34.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:45:34.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:34.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:45:34.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:45:34.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:45:34.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:45:34.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:45:34.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:45:34.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:45:34.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:34.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:45:34.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154105 2024-04-25T07:45:34.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:45:34.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:45:34.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:45:34.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:45:34.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:34.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:45:34.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.154105 2024-04-25T07:45:34.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_config_track td/osd-config 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:54: TEST_config_track: local dir=td/osd-config 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:56: TEST_config_track: run_mon td/osd-config a 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-config 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:45:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:45:34.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-config/a 2024-04-25T07:45:34.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-config/a --run-dir=td/osd-config 2024-04-25T07:45:35.378 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:45:35.378 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:45:35.378 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154105/$cluster-$name.asok' 2024-04-25T07:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-config/a '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' --mon-cluster-log-file=td/osd-config/log --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:45:35.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:45:35.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:45:35.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:45:35.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:45:35.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:45:35.633 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154105/ceph-mon.a.asok 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:45:35.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154105/ceph-mon.a.asok config get fsid 2024-04-25T07:45:35.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:45:35.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:45:35.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:45:35.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:45:35.748 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:45:35.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:45:35.748 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:45:35.748 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:45:35.748 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:45:35.749 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:35.749 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:45:35.749 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154105/ceph-mon.a.asok 2024-04-25T07:45:35.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:45:35.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.154105/ceph-mon.a.asok config get mon_host 2024-04-25T07:45:35.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:57: TEST_config_track: run_mgr td/osd-config x 2024-04-25T07:45:35.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-config 2024-04-25T07:45:35.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:45:35.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:45:35.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:45:35.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-config/x 2024-04-25T07:45:35.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:45:36.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:45:36.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:45:36.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:45:36.814 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:45:36.814 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:36.814 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:45:36.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154105/$cluster-$name.asok' 2024-04-25T07:45:36.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:45:36.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-config/x '--log-file=td/osd-config/$name.log' '--admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' --run-dir=td/osd-config '--pid-file=td/osd-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:45:36.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:58: TEST_config_track: run_osd td/osd-config 0 2024-04-25T07:45:36.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-config 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-config/0 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=00c37098-7115-49ca-8673-1548f53aa490 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-config/0' 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-config/0/journal' 2024-04-25T07:45:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-config' 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.154105/$cluster-$name.asok' 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:45:36.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:45:36.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-config/$name.log' 2024-04-25T07:45:36.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-config/$name.pid' 2024-04-25T07:45:36.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:45:36.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:45:36.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:45:36.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:45:36.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:45:36.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:45:36.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-config/0 2024-04-25T07:45:36.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:45:36.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e3a7c8a7-247e-460f-a4d5-e73a5dfe12ed 2024-04-25T07:45:36.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e3a7c8a7-247e-460f-a4d5-e73a5dfe12ed' 2024-04-25T07:45:36.857 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 e3a7c8a7-247e-460f-a4d5-e73a5dfe12ed 2024-04-25T07:45:36.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:45:36.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgCipmgdtUNBAAPW3TqMDOK/ugAxiMTJqHrA== 2024-04-25T07:45:36.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgCipmgdtUNBAAPW3TqMDOK/ugAxiMTJqHrA=="}' 2024-04-25T07:45:36.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e3a7c8a7-247e-460f-a4d5-e73a5dfe12ed -i td/osd-config/0/new.json 2024-04-25T07:45:37.235 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:45:37.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-config/0/new.json 2024-04-25T07:45:37.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=00c37098-7115-49ca-8673-1548f53aa490 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgCipmgdtUNBAAPW3TqMDOK/ugAxiMTJqHrA== --osd-uuid e3a7c8a7-247e-460f-a4d5-e73a5dfe12ed 2024-04-25T07:45:37.284 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:37.282+0000 7fbd53c647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:37.337 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:37.334+0000 7fbd53c647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:37.339 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:37.338+0000 7fbd53c647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:37.339 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:37.338+0000 7fbd53c647c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:45:37.339 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:37.338+0000 7fbd53c647c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:45:37.339 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:37.338+0000 7fbd53c647c0 -1 bluestore(td/osd-config/0/block) _read_bdev_label failed to open td/osd-config/0/block: (2) No such file or directory 2024-04-25T07:45:37.376 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:37.374+0000 7fbd53c647c0 -1 bluestore(td/osd-config/0) _read_fsid unparsable uuid 2024-04-25T07:45:45.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-config/0/keyring 2024-04-25T07:45:45.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:45:45.142 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:45:45.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:45:45.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:45:45.704 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:45:45.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:45:45.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=00c37098-7115-49ca-8673-1548f53aa490 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-config/0 --osd-journal=td/osd-config/0/journal --chdir= --run-dir=td/osd-config '--admin-socket=/tmp/ceph-asok.154105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-config/$name.log' '--pid-file=td/osd-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:45:45.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:45:45.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:45:45.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:45:45.734 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:45.737+0000 7fe96b2ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:45.736 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:45.737+0000 7fe96b2ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:45.737 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:45.741+0000 7fe96b2ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:45:46.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:45:46.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:45:46.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:45:46.114 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:45:46.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:45:46.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:45:46.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:46.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:45:46.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:46.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:46.299 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:46.301+0000 7fe96b2ee7c0 -1 Falling back to public interface 2024-04-25T07:45:46.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:47.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:47.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:47.512 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:45:47.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:45:47.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:47.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:47.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:48.809 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:45:48.813+0000 7fe96b2ee7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:45:48.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:48.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:48.939 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:45:48.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:45:48.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:48.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:49.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:50.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:50.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:50.670 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:45:50.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:45:50.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:50.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:51.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:52.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:52.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:52.227 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:45:52.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:45:52.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:52.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:52.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:53.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:53.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:53.821 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:45:53.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:45:53.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:53.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:54.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:55.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:55.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:55.251 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:45:55.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:45:55.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:55.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:55.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:56.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:56.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:56.819 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:45:56.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:45:56.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:56.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:57.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:58.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:58.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:58.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:45:58.262 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:45:58.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:58.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:45:58.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:45:59.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:45:59.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:45:59.880 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:45:59.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:45:59.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:45:59.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:00.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:01.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:01.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:01.299 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:46:01.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:46:01.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:01.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:01.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:02.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:02.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:02.791 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:46:02.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:46:02.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:02.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:04.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:04.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:04.282 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:46:04.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:46:04.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:04.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:04.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:05.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:05.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:05.834 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:46:05.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:46:05.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:05.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:06.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:07.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:07.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:07.397 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:46:07.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:46:07.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:07.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:08.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:09.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:09.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:09.897 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:46:09.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:46:09.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:09.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:11.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:12.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:12.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:12.328 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:46:12.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:46:12.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:12.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:13.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:14.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:14.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:14.475 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:46:14.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:46:14.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:14.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:15.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:16.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:16.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:16.539 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:46:16.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:46:16.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:16.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:17.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:18.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:18.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:18.500 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:46:18.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:46:18.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:18.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:19.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:20.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:20.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:20.281 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:46:20.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:46:20.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:20.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:20.691 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2262307704,v1:127.0.0.1:6803/2262307704] [v2:127.0.0.1:6804/2262307704,v1:127.0.0.1:6805/2262307704] exists,up e3a7c8a7-247e-460f-a4d5-e73a5dfe12ed 2024-04-25T07:46:20.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:46:20.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:46:20.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:46:20.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: CEPH_ARGS= 2024-04-25T07:46:20.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: ceph-conf --show-config-value osd_map_cache_size 2024-04-25T07:46:20.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:61: TEST_config_track: local osd_map_cache_size=50 2024-04-25T07:46:20.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: CEPH_ARGS= 2024-04-25T07:46:20.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: ceph-conf --show-config-value osd_pg_epoch_persisted_max_stale 2024-04-25T07:46:20.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:63: TEST_config_track: local osd_pg_epoch_persisted_max_stale=40 2024-04-25T07:46:20.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:68: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:46:20.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:69: TEST_config_track: local stale=100 2024-04-25T07:46:20.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:70: TEST_config_track: ceph tell osd.0 injectargs '--osd-pg-epoch-persisted-max-stale 100' 2024-04-25T07:46:20.882 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T07:46:20.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: get_asok_path osd.0 2024-04-25T07:46:20.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:46:20.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:46:20.898 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:46:20.898 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:20.898 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:46:20.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154105/ceph-osd.0.asok 2024-04-25T07:46:20.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: CEPH_ARGS= 2024-04-25T07:46:20.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:71: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.154105/ceph-osd.0.asok log flush 2024-04-25T07:46:20.995 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T07:46:21.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:72: TEST_config_track: grep 'is not > osd_pg_epoch_persisted_max_stale' td/osd-config/osd.0.log 2024-04-25T07:46:21.009 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T07:46:20.873+0000 7fe969aad640 0 log_channel(cluster) log [WRN] : osd_map_cache_size (50) is not > osd_pg_epoch_persisted_max_stale (100) 2024-04-25T07:46:21.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:73: TEST_config_track: rm td/osd-config/osd.0.log 2024-04-25T07:46:21.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: get_asok_path osd.0 2024-04-25T07:46:21.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T07:46:21.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T07:46:21.011 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:46:21.011 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:21.012 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:46:21.012 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.154105/ceph-osd.0.asok 2024-04-25T07:46:21.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: CEPH_ARGS= 2024-04-25T07:46:21.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:74: TEST_config_track: ceph --admin-daemon /tmp/ceph-asok.154105/ceph-osd.0.asok log reopen 2024-04-25T07:46:21.129 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T07:46:21.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:46:21.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:46:21.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:46:21.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:46:21.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:21.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:21.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:21.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:21.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:21.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:21.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:21.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:21.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:21.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:21.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:21.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:21.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:21.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:21.460 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:21.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:46:21.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:46:21.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:21.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:21.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:46:21.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154105 2024-04-25T07:46:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:31: run: for func in $funcs 2024-04-25T07:46:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:32: run: setup td/osd-config 2024-04-25T07:46:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-config 2024-04-25T07:46:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-config 2024-04-25T07:46:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:46:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:46:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:46:21.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:21.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:21.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:21.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:21.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:21.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:21.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:21.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:21.478 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:21.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:21.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:21.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:21.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:21.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:21.481 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:21.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:46:21.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:46:21.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:21.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:21.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:46:21.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154105 2024-04-25T07:46:21.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:21.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:21.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-config 2024-04-25T07:46:21.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:46:21.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:21.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:46:21.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.154105 2024-04-25T07:46:21.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:46:21.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:46:21.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:46:21.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-config 1' TERM HUP INT 2024-04-25T07:46:21.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:33: run: TEST_default_adjustment td/osd-config 2024-04-25T07:46:21.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin 2024-04-25T07:46:21.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:78: TEST_default_adjustment: a= 2024-04-25T07:46:21.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin default 2024-04-25T07:46:21.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:79: TEST_default_adjustment: b=default 2024-04-25T07:46:21.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value rgw_torrent_origin --default-rgw-torrent-origin arg 2024-04-25T07:46:21.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:80: TEST_default_adjustment: c=arg 2024-04-25T07:46:21.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:81: TEST_default_adjustment: '[' '' '!=' default ']' 2024-04-25T07:46:21.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:82: TEST_default_adjustment: '[' default = default ']' 2024-04-25T07:46:21.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:83: TEST_default_adjustment: '[' arg = arg ']' 2024-04-25T07:46:21.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file 2024-04-25T07:46:21.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:85: TEST_default_adjustment: a=true 2024-04-25T07:46:21.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false 2024-04-25T07:46:21.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:86: TEST_default_adjustment: b=false 2024-04-25T07:46:21.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: ceph-osd --no-mon-config --show-config-value log_to_file --default-log-to-file=false --log-to-file 2024-04-25T07:46:21.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:87: TEST_default_adjustment: c=true 2024-04-25T07:46:21.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:88: TEST_default_adjustment: '[' true = true ']' 2024-04-25T07:46:21.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:89: TEST_default_adjustment: '[' false = false ']' 2024-04-25T07:46:21.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:90: TEST_default_adjustment: '[' true = true ']' 2024-04-25T07:46:21.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-config.sh:34: run: teardown td/osd-config 2024-04-25T07:46:21.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:46:21.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:46:21.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:46:21.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:21.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:21.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:21.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:21.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:21.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:21.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:21.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:21.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:21.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:21.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:21.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:21.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:21.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:21.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:21.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:46:21.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:46:21.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:21.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:21.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:46:21.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154105 2024-04-25T07:46:21.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:21.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:21.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:46:21.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-config 0 2024-04-25T07:46:21.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-config 2024-04-25T07:46:21.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:46:21.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-config KILL 2024-04-25T07:46:21.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:21.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:21.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:21.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:21.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:21.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:21.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:21.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:21.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:21.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:21.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:21.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:21.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:21.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:21.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:21.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:46:21.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-config 2024-04-25T07:46:21.638 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:21.638 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:21.638 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.154105 2024-04-25T07:46:21.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.154105 2024-04-25T07:46:21.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:21.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:21.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:46:21.640 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:46:21.640 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:46:21.693 INFO:tasks.workunit:Running workunit osd/osd-copy-from.sh... 2024-04-25T07:46:21.694 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-copy-from.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh 2024-04-25T07:46:21.748 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:46:21.754 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:46:21.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:46:21.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:46:21.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:46:21.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:46:21.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:46:21.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:46:21.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:46:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:46:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-copy-from 2024-04-25T07:46:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:23: run: local dir=td/osd-copy-from 2024-04-25T07:46:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:24: run: shift 2024-04-25T07:46:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: export CEPH_MON=127.0.0.1:7111 2024-04-25T07:46:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:26: run: CEPH_MON=127.0.0.1:7111 2024-04-25T07:46:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:27: run: export CEPH_ARGS 2024-04-25T07:46:21.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: uuidgen 2024-04-25T07:46:21.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:28: run: CEPH_ARGS+='--fsid=43a06666-10e2-4f1a-a296-980235d2a6d4 --auth-supported=none ' 2024-04-25T07:46:21.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7111 ' 2024-04-25T07:46:21.756 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: set 2024-04-25T07:46:21.756 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:31: run: local funcs=TEST_copy_from 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:32: run: for func in $funcs 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:33: run: setup td/osd-copy-from 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-copy-from 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-copy-from 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:46:21.758 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:46:21.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:46:21.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:46:21.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:46:21.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:46:21.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:46:21.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:46:21.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:46:21.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:46:21.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:46:21.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:46:21.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:46:21.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:46:21.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:46:21.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:46:21.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:46:21.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:46:21.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:46:21.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:21.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:46:21.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.158424 2024-04-25T07:46:21.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:46:21.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:46:21.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-copy-from 2024-04-25T07:46:21.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:46:21.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:21.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:46:21.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.158424 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-copy-from 1' TERM HUP INT 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:34: run: TEST_copy_from td/osd-copy-from 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:40: TEST_copy_from: local dir=td/osd-copy-from 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:42: TEST_copy_from: run_mon td/osd-copy-from a 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-copy-from 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:46:21.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-copy-from/a 2024-04-25T07:46:21.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-copy-from/a --run-dir=td/osd-copy-from 2024-04-25T07:46:22.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:46:22.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:22.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:22.543 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:22.543 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:22.543 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:46:22.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158424/$cluster-$name.asok' 2024-04-25T07:46:22.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-copy-from/a '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.158424/$cluster-$name.asok' --mon-cluster-log-file=td/osd-copy-from/log --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:46:22.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:46:22.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:46:22.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:46:22.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:46:22.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:46:22.803 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:46:22.803 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:46:22.803 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:46:22.803 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:46:22.803 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:46:22.803 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:22.803 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:46:22.803 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.158424/ceph-mon.a.asok 2024-04-25T07:46:22.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:46:22.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.158424/ceph-mon.a.asok config get fsid 2024-04-25T07:46:22.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:46:22.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:46:22.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:46:22.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:46:22.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:46:22.927 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:46:22.927 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:46:22.927 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:46:22.927 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:46:22.927 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:22.927 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:46:22.927 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.158424/ceph-mon.a.asok 2024-04-25T07:46:22.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:46:22.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.158424/ceph-mon.a.asok config get mon_host 2024-04-25T07:46:23.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:43: TEST_copy_from: run_mgr td/osd-copy-from x 2024-04-25T07:46:23.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-copy-from 2024-04-25T07:46:23.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:46:23.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:46:23.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:46:23.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-copy-from/x 2024-04-25T07:46:23.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:46:23.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:46:23.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:23.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:23.963 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:23.963 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:23.963 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:46:23.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158424/$cluster-$name.asok' 2024-04-25T07:46:23.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:46:23.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-copy-from/x '--log-file=td/osd-copy-from/$name.log' '--admin-socket=/tmp/ceph-asok.158424/$cluster-$name.asok' --run-dir=td/osd-copy-from '--pid-file=td/osd-copy-from/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:44: TEST_copy_from: run_osd td/osd-copy-from 0 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/0 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43a06666-10e2-4f1a-a296-980235d2a6d4 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:46:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/0' 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/0/journal' 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158424/$cluster-$name.asok' 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.158424/$cluster-$name.asok' 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:46:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/0 2024-04-25T07:46:24.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:46:24.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9bc144e-a5fc-48d8-a620-7ff23b1cb037 2024-04-25T07:46:24.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d9bc144e-a5fc-48d8-a620-7ff23b1cb037' 2024-04-25T07:46:24.006 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 d9bc144e-a5fc-48d8-a620-7ff23b1cb037 2024-04-25T07:46:24.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:46:24.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQCipm3Ml+ARAA74AE0Iw1JqU41dgo2vrOnQ== 2024-04-25T07:46:24.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQCipm3Ml+ARAA74AE0Iw1JqU41dgo2vrOnQ=="}' 2024-04-25T07:46:24.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9bc144e-a5fc-48d8-a620-7ff23b1cb037 -i td/osd-copy-from/0/new.json 2024-04-25T07:46:24.445 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:46:24.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/0/new.json 2024-04-25T07:46:24.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=43a06666-10e2-4f1a-a296-980235d2a6d4 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQCipm3Ml+ARAA74AE0Iw1JqU41dgo2vrOnQ== --osd-uuid d9bc144e-a5fc-48d8-a620-7ff23b1cb037 2024-04-25T07:46:24.498 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:24.500+0000 7f02b04667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:24.548 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:24.548+0000 7f02b04667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:24.550 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:24.552+0000 7f02b04667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:24.550 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:24.552+0000 7f02b04667c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:46:24.550 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:24.552+0000 7f02b04667c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:46:24.551 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:24.552+0000 7f02b04667c0 -1 bluestore(td/osd-copy-from/0/block) _read_bdev_label failed to open td/osd-copy-from/0/block: (2) No such file or directory 2024-04-25T07:46:24.608 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:24.608+0000 7f02b04667c0 -1 bluestore(td/osd-copy-from/0) _read_fsid unparsable uuid 2024-04-25T07:46:32.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/0/keyring 2024-04-25T07:46:32.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:46:32.394 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:46:32.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:46:32.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:46:32.945 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:46:32.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:46:32.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:46:32.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=43a06666-10e2-4f1a-a296-980235d2a6d4 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/0 --osd-journal=td/osd-copy-from/0/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:46:32.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:46:32.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:46:32.974 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:32.976+0000 7f2afc6457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:32.977 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:32.976+0000 7f2afc6457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:32.978 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:32.980+0000 7f2afc6457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:46:33.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:46:33.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:46:33.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:46:33.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:46:33.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:46:33.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:33.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:46:33.338 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:46:33.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:33.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:33.544 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:33.544+0000 7f2afc6457c0 -1 Falling back to public interface 2024-04-25T07:46:33.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:34.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:34.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:34.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:46:34.732 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:46:34.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:34.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:35.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:36.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:36.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:36.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:46:36.143 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:46:36.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:36.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:36.356 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:46:36.356+0000 7f2afc6457c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:46:36.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:37.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:37.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:37.612 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:46:37.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:46:37.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:37.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:38.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:39.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:39.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:39.057 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:46:39.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:46:39.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:39.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:39.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:40.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:40.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:40.542 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:46:40.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:46:40.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:40.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:41.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:42.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:42.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:42.007 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:46:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:46:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:42.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:42.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:43.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:43.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:43.544 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:46:43.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:46:43.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:43.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:44.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:45.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:45.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:45.032 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:46:45.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:46:45.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:45.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:45.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:46.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:46.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:46.594 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:46:46.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:46:46.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:46.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:47.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:48.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:48.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:48.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:46:48.078 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:46:48.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:48.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:48.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:49.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:49.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:49.569 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:46:49.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:46:49.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:49.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:50.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:51.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:51.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:51.140 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:46:51.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:46:51.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:51.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:51.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:52.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:52.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:52.626 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:46:52.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:46:52.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:52.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:53.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:46:54.135 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:46:54.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:54.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:54.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:55.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:55.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:55.758 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:46:55.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:46:55.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:55.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:56.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:57.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:57.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:57.627 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:46:57.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:46:57.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:57.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:46:58.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:46:59.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:46:59.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:46:59.422 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:46:59.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:46:59.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:46:59.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:00.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:01.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:01.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:01.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:47:01.631 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:47:01.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:01.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:02.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:03.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:03.527 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:47:03.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:47:03.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:03.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:04.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:05.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:05.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:05.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:47:05.257 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:47:05.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:05.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:06.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:07.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:07.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:07.097 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:47:07.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:47:07.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:07.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:07.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:08.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:08.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:08.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:47:08.678 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:47:08.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:08.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:47:09.138 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4093307104,v1:127.0.0.1:6803/4093307104] [v2:127.0.0.1:6804/4093307104,v1:127.0.0.1:6805/4093307104] exists,up d9bc144e-a5fc-48d8-a620-7ff23b1cb037 2024-04-25T07:47:09.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:47:09.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:45: TEST_copy_from: run_osd td/osd-copy-from 1 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-copy-from 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-copy-from/1 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=43a06666-10e2-4f1a-a296-980235d2a6d4 --auth-supported=none --mon-host=127.0.0.1:7111 ' 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-copy-from/1' 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-copy-from/1/journal' 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:47:09.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-copy-from' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.158424/$cluster-$name.asok' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.158424/$cluster-$name.asok' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-copy-from/$name.log' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-copy-from/$name.pid' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:47:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:47:09.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:47:09.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:47:09.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:47:09.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:47:09.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-copy-from/1 2024-04-25T07:47:09.145 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:47:09.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=692cd164-a78d-4f28-b697-f0343782b47e 2024-04-25T07:47:09.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 692cd164-a78d-4f28-b697-f0343782b47e' 2024-04-25T07:47:09.146 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 692cd164-a78d-4f28-b697-f0343782b47e 2024-04-25T07:47:09.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:47:09.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB9CipmpGQEChAArs0uEL5hLVmN2k/8Ok4gnA== 2024-04-25T07:47:09.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB9CipmpGQEChAArs0uEL5hLVmN2k/8Ok4gnA=="}' 2024-04-25T07:47:09.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 692cd164-a78d-4f28-b697-f0343782b47e -i td/osd-copy-from/1/new.json 2024-04-25T07:47:09.722 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:47:09.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-copy-from/1/new.json 2024-04-25T07:47:09.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=43a06666-10e2-4f1a-a296-980235d2a6d4 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9CipmpGQEChAArs0uEL5hLVmN2k/8Ok4gnA== --osd-uuid 692cd164-a78d-4f28-b697-f0343782b47e 2024-04-25T07:47:09.770 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:09.771+0000 7fb4e3fd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:09.772 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:09.771+0000 7fb4e3fd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:09.773 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:09.775+0000 7fb4e3fd67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:09.774 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:09.775+0000 7fb4e3fd67c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:47:09.774 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:09.775+0000 7fb4e3fd67c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:47:09.774 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:09.775+0000 7fb4e3fd67c0 -1 bluestore(td/osd-copy-from/1/block) _read_bdev_label failed to open td/osd-copy-from/1/block: (2) No such file or directory 2024-04-25T07:47:09.830 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:09.831+0000 7fb4e3fd67c0 -1 bluestore(td/osd-copy-from/1) _read_fsid unparsable uuid 2024-04-25T07:47:18.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-copy-from/1/keyring 2024-04-25T07:47:18.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:47:18.116 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:47:18.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:47:18.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-copy-from/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:47:18.663 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:47:18.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:47:18.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=43a06666-10e2-4f1a-a296-980235d2a6d4 --auth-supported=none --mon-host=127.0.0.1:7111 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-copy-from/1 --osd-journal=td/osd-copy-from/1/journal --chdir= --run-dir=td/osd-copy-from '--admin-socket=/tmp/ceph-asok.158424/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-copy-from/$name.log' '--pid-file=td/osd-copy-from/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:47:18.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:47:18.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:47:18.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:47:18.692 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:18.691+0000 7f84f37ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:18.695 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:18.695+0000 7f84f37ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:18.696 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:18.695+0000 7f84f37ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:47:19.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:47:19.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:47:19.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:47:19.081 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:47:19.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:47:19.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:47:19.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:19.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:47:19.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:19.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:19.251 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:19.251+0000 7f84f37ba7c0 -1 Falling back to public interface 2024-04-25T07:47:19.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:20.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:20.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:20.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:47:20.551 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:47:20.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:20.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:20.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:21.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:21.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:21.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:47:21.969 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:47:21.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:21.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:22.031 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:47:22.031+0000 7f84f37ba7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:47:22.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:23.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:23.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:23.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:47:23.524 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:47:23.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:23.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:24.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:25.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:25.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:25.116 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:47:25.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:47:25.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:25.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:25.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:26.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:26.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:26.718 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:47:26.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:47:26.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:26.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:27.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:28.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:28.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:28.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:47:28.263 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:47:28.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:28.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:28.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:29.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:29.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:29.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:47:29.691 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:47:29.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:29.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:30.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:31.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:31.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:31.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:47:31.451 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:47:31.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:31.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:32.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:33.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:33.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:33.157 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:47:33.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:47:33.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:33.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:33.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:34.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:34.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:34.749 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:47:34.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:47:34.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:34.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:35.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:36.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:36.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:36.269 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:47:36.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:47:36.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:36.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:36.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:37.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:37.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:37.697 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:47:37.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:47:37.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:37.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:38.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:39.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:39.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:39.318 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:47:39.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:47:39.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:39.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:39.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:40.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:40.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:40.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:47:40.921 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:47:40.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:40.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:41.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:42.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:42.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:42.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:47:42.783 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:47:42.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:42.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:43.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:44.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:44.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:44.824 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:47:44.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:47:44.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:44.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:45.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:46.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:46.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:46.456 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:47:46.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:47:46.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:46.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:47.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:48.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:48.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:48.763 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:47:48.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:47:48.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:48.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:49.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:50.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:47:50.875 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:47:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:51.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:52.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:52.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:52.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:47:52.996 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:47:52.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:52.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:53.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:47:54.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:47:54.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:47:54.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:47:54.730 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:47:54.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:47:54.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:47:55.147 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3106463853,v1:127.0.0.1:6811/3106463853] [v2:127.0.0.1:6812/3106463853,v1:127.0.0.1:6813/3106463853] exists,up 692cd164-a78d-4f28-b697-f0343782b47e 2024-04-25T07:47:55.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:47:55.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:47:55.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:47:55.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:46: TEST_copy_from: create_rbd_pool 2024-04-25T07:47:55.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:47:55.547 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' does not exist 2024-04-25T07:47:55.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:47:55.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:47:56.245 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' already exists 2024-04-25T07:47:56.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:47:57.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:47:59.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: which rados 2024-04-25T07:47:59.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:49: TEST_copy_from: rados -p rbd put foo /usr/bin/rados 2024-04-25T07:47:59.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:50: TEST_copy_from: rados -p rbd cp foo foo2 2024-04-25T07:48:00.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:51: TEST_copy_from: rados -p rbd stat foo2 2024-04-25T07:48:00.210 INFO:tasks.workunit.client.0.smithi134.stdout:rbd/foo2 mtime 2024-04-25T07:47:59.000000+0000, size 572320 2024-04-25T07:48:00.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:54: TEST_copy_from: ceph tell 'osd.*' injectargs -- --osd-debug-inject-copyfrom-error 2024-04-25T07:48:00.352 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T07:48:00.380 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T07:48:00.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:55: TEST_copy_from: rados -p rbd cp foo foo3 2024-04-25T07:48:00.450 INFO:tasks.workunit.client.0.smithi134.stderr:error copying rbd/foo => rbd/foo3: (5) Input/output error 2024-04-25T07:48:00.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:56: TEST_copy_from: rados -p rbd stat foo3 2024-04-25T07:48:00.514 INFO:tasks.workunit.client.0.smithi134.stderr: error stat-ing rbd/foo3: (2) No such file or directory 2024-04-25T07:48:00.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:59: TEST_copy_from: ceph tell 'osd.*' injectargs -- --no-osd-debug-inject-copyfrom-error 2024-04-25T07:48:00.647 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T07:48:00.677 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T07:48:00.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:60: TEST_copy_from: rados -p rbd cp foo foo3 2024-04-25T07:48:01.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:61: TEST_copy_from: rados -p rbd stat foo3 2024-04-25T07:48:01.055 INFO:tasks.workunit.client.0.smithi134.stdout:rbd/foo3 mtime 2024-04-25T07:47:59.000000+0000, size 572320 2024-04-25T07:48:01.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-copy-from.sh:35: run: teardown td/osd-copy-from 2024-04-25T07:48:01.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:48:01.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:48:01.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:48:01.058 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:01.058 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:01.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:01.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:01.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:01.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:01.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:48:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:48:01.167 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:48:01.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:48:01.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:48:01.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:48:01.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:01.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:48:01.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:48:01.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:01.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:48:01.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:48:01.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:48:01.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:48:01.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:01.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:48:01.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.158424 2024-04-25T07:48:01.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:48:01.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:48:01.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:48:01.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-copy-from 0 2024-04-25T07:48:01.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-copy-from 2024-04-25T07:48:01.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:48:01.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-copy-from KILL 2024-04-25T07:48:01.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:01.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:01.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:01.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:01.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:01.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:01.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:48:01.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:48:01.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:48:01.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:48:01.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:48:01.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:48:01.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:01.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:48:01.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:48:01.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:01.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:48:01.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:48:01.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-copy-from 2024-04-25T07:48:01.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:48:01.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:01.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.158424 2024-04-25T07:48:01.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.158424 2024-04-25T07:48:01.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:48:01.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:48:01.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:48:01.194 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:48:01.194 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:48:01.246 INFO:tasks.workunit:Running workunit osd/osd-dup.sh... 2024-04-25T07:48:01.246 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-dup.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh 2024-04-25T07:48:01.300 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:48:01.305 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:48:01.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-dup 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:8: run: local dir=td/osd-dup 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:9: run: shift 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: export CEPH_MON=127.0.0.1:7146 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:11: run: CEPH_MON=127.0.0.1:7146 2024-04-25T07:48:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:12: run: export CEPH_ARGS 2024-04-25T07:48:01.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: uuidgen 2024-04-25T07:48:01.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:13: run: CEPH_ARGS+='--fsid=1ba05121-3379-40e9-a70b-83ffe7ef23b6 --auth-supported=none ' 2024-04-25T07:48:01.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-04-25T07:48:01.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:16: run: CEPH_ARGS+='--filestore_wbthrottle_xfs_ios_hard_limit=900 ' 2024-04-25T07:48:01.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:17: run: CEPH_ARGS+='--filestore_wbthrottle_btrfs_ios_hard_limit=900 ' 2024-04-25T07:48:01.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: set 2024-04-25T07:48:01.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:48:01.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-dup.sh:18: run: local funcs= 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-dup 0 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-dup 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-dup KILL 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:01.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:01.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:01.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:48:01.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:48:01.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:48:01.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:48:01.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:48:01.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:48:01.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:01.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:48:01.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:48:01.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:01.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:48:01.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:48:01.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-dup 2024-04-25T07:48:01.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:48:01.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:01.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162682 2024-04-25T07:48:01.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162682 2024-04-25T07:48:01.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:48:01.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:48:01.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:48:01.318 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:48:01.318 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:48:01.370 INFO:tasks.workunit:Running workunit osd/osd-fast-mark-down.sh... 2024-04-25T07:48:01.370 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-fast-mark-down.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh 2024-04-25T07:48:01.423 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:48:01.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-fast-mark-down 2024-04-25T07:48:01.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:23: run: local dir=td/osd-fast-mark-down 2024-04-25T07:48:01.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:24: run: shift 2024-04-25T07:48:01.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:25: run: rm -f 'td/osd-fast-mark-down/*.pid' 2024-04-25T07:48:01.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: export CEPH_MON=127.0.0.1:7126 2024-04-25T07:48:01.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:26: run: CEPH_MON=127.0.0.1:7126 2024-04-25T07:48:01.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:27: run: export CEPH_ARGS 2024-04-25T07:48:01.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: uuidgen 2024-04-25T07:48:01.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:28: run: CEPH_ARGS+='--fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none ' 2024-04-25T07:48:01.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:30: run: OLD_ARGS='--fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none ' 2024-04-25T07:48:01.430 INFO:tasks.workunit.client.0.smithi134.stdout:Ensuring old behavior is there... 2024-04-25T07:48:01.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:31: run: CEPH_ARGS+='--osd-fast-fail-on-connection-refused=false ' 2024-04-25T07:48:01.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:32: run: echo 'Ensuring old behavior is there...' 2024-04-25T07:48:01.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:33: run: test_fast_kill td/osd-fast-mark-down 2024-04-25T07:48:01.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-04-25T07:48:01.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-04-25T07:48:01.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-04-25T07:48:01.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:48:01.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:48:01.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:48:01.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:01.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:01.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:01.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:01.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:01.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:01.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:48:01.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:48:01.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:48:01.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:48:01.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:48:01.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:48:01.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:01.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:48:01.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:48:01.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:01.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:48:01.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:48:01.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:48:01.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:48:01.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:01.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:01.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162711 2024-04-25T07:48:01.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:48:01.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:48:01.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-04-25T07:48:01.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:48:01.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:01.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:01.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162711 2024-04-25T07:48:01.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:48:01.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:48:01.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:48:01.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:48:01.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-04-25T07:48:01.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-04-25T07:48:01.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-04-25T07:48:01.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:48:01.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:48:01.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:48:01.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-04-25T07:48:01.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-04-25T07:48:01.473 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:01.470+0000 7f82f6870c80 -1 failed for service _ceph-mon._tcp 2024-04-25T07:48:01.473 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:01.470+0000 7f82f6870c80 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-25T07:48:01.473 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T07:48:01.473 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:01.470+0000 7f82f6870c80 -1 ceph-mon: warning: no initial monitors; must use admin socket to feed hints 2024-04-25T07:48:01.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:48:01.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:01.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:01.946 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:01.947 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:01.947 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:01.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162711/$cluster-$name.asok' 2024-04-25T07:48:01.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T07:48:02.167 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:02.166+0000 7f3cb26b3c80 -1 failed for service _ceph-mon._tcp 2024-04-25T07:48:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:02.166+0000 7f3cb26b3c80 -1 unable to get monitor info from DNS SRV with service name: ceph-mon 2024-04-25T07:48:02.168 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T07:48:02.168 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:02.166+0000 7f3cb26b3c80 -1 ceph-mon: error generating initial monmap: (2) No such file or directory 2024-04-25T07:48:02.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: return 1 2024-04-25T07:48:02.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: return 1 2024-04-25T07:48:02.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:35: run: CEPH_ARGS='--fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-04-25T07:48:02.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:36: run: OLD_ARGS='--fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true ' 2024-04-25T07:48:02.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:38: run: CEPH_ARGS='--fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:48:02.176 INFO:tasks.workunit.client.0.smithi134.stdout:Testing async msgr... 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:39: run: echo 'Testing async msgr...' 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:40: run: test_fast_kill td/osd-fast-mark-down 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:48: test_fast_kill: setup td/osd-fast-mark-down 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-fast-mark-down 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-fast-mark-down 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:48:02.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:48:02.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:48:02.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:48:02.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:48:02.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:48:02.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:48:02.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:48:02.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:48:02.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:02.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:48:02.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:48:02.185 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:48:02.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:48:02.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:48:02.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:48:02.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:48:02.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:02.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:02.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162711 2024-04-25T07:48:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:48:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:48:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-fast-mark-down 2024-04-25T07:48:02.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:48:02.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:02.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:02.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.162711 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-fast-mark-down 1' TERM HUP INT 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:49: test_fast_kill: run_mon td/osd-fast-mark-down a --osd_pool_default_size=3 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-fast-mark-down 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-fast-mark-down/a 2024-04-25T07:48:02.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-fast-mark-down/a --run-dir=td/osd-fast-mark-down --osd_pool_default_size=3 2024-04-25T07:48:02.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:48:02.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:02.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:02.707 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:02.707 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:02.708 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:02.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162711/$cluster-$name.asok' 2024-04-25T07:48:02.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-fast-mark-down/a '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' --mon-cluster-log-file=td/osd-fast-mark-down/log --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T07:48:02.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:48:02.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:48:02.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:48:02.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:48:02.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:48:02.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:48:02.986 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:48:02.986 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:48:02.986 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:48:02.986 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:48:02.987 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:02.987 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:02.987 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162711/ceph-mon.a.asok 2024-04-25T07:48:02.987 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:48:02.987 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162711/ceph-mon.a.asok config get fsid 2024-04-25T07:48:03.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:48:03.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:48:03.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:48:03.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:48:03.100 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:48:03.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:48:03.100 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:48:03.100 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:48:03.100 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:48:03.100 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:03.100 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:03.101 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.162711/ceph-mon.a.asok 2024-04-25T07:48:03.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:48:03.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.162711/ceph-mon.a.asok config get mon_host 2024-04-25T07:48:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:50: test_fast_kill: run_mgr td/osd-fast-mark-down x 2024-04-25T07:48:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-fast-mark-down 2024-04-25T07:48:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:48:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:48:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:48:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-fast-mark-down/x 2024-04-25T07:48:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:48:04.160 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:48:04.160 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:04.160 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:04.161 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:04.161 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:04.161 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:04.161 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162711/$cluster-$name.asok' 2024-04-25T07:48:04.161 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:48:04.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-fast-mark-down/x '--log-file=td/osd-fast-mark-down/$name.log' '--admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' --run-dir=td/osd-fast-mark-down '--pid-file=td/osd-fast-mark-down/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:48:04.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:48:04.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 0 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/0 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/0' 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/0/journal' 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:48:04.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162711/$cluster-$name.asok' 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:48:04.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:48:04.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:48:04.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:48:04.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:48:04.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:48:04.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:48:04.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:48:04.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/0 2024-04-25T07:48:04.246 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:48:04.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c49c788-554d-4172-a2c9-0819d6bd8807 2024-04-25T07:48:04.247 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 9c49c788-554d-4172-a2c9-0819d6bd8807 2024-04-25T07:48:04.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9c49c788-554d-4172-a2c9-0819d6bd8807' 2024-04-25T07:48:04.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:48:04.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC0CipmfL4CEBAAPEdS98bVzFBctYuWQgCMCQ== 2024-04-25T07:48:04.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC0CipmfL4CEBAAPEdS98bVzFBctYuWQgCMCQ=="}' 2024-04-25T07:48:04.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c49c788-554d-4172-a2c9-0819d6bd8807 -i td/osd-fast-mark-down/0/new.json 2024-04-25T07:48:04.631 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:48:04.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/0/new.json 2024-04-25T07:48:04.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC0CipmfL4CEBAAPEdS98bVzFBctYuWQgCMCQ== --osd-uuid 9c49c788-554d-4172-a2c9-0819d6bd8807 2024-04-25T07:48:04.685 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:04.682+0000 7feec06e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:04.726 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:04.726+0000 7feec06e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:04.727 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:04.726+0000 7feec06e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:04.728 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:04.726+0000 7feec06e57c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:48:04.728 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:04.726+0000 7feec06e57c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:48:04.728 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:04.726+0000 7feec06e57c0 -1 bluestore(td/osd-fast-mark-down/0/block) _read_bdev_label failed to open td/osd-fast-mark-down/0/block: (2) No such file or directory 2024-04-25T07:48:04.772 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:04.770+0000 7feec06e57c0 -1 bluestore(td/osd-fast-mark-down/0) _read_fsid unparsable uuid 2024-04-25T07:48:12.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/0/keyring 2024-04-25T07:48:12.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:48:12.498 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:48:12.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:48:12.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:48:13.073 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:48:13.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:48:13.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:48:13.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:48:13.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:48:13.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/0 --osd-journal=td/osd-fast-mark-down/0/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:48:13.108 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:13.106+0000 7f7665a6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:13.110 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:13.110+0000 7f7665a6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:13.111 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:13.110+0000 7f7665a6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:13.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:48:13.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:48:13.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:48:13.483 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:48:13.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:48:13.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:48:13.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:13.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:48:13.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:13.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:13.663 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:13.662+0000 7f7665a6c7c0 -1 Falling back to public interface 2024-04-25T07:48:13.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:14.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:14.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:14.905 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:48:14.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:48:14.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:14.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:15.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:16.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:16.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:16.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:48:16.322 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:48:16.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:16.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:16.499 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:16.498+0000 7f7665a6c7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:48:16.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:17.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:17.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:17.757 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:48:17.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:48:17.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:17.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:18.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:19.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:19.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:19.185 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:48:19.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:48:19.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:19.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:19.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:20.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:20.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:20.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:48:20.697 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:48:20.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:20.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:21.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:22.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:22.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:22.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:48:22.135 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:48:22.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:22.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:22.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:23.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:23.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:23.647 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:48:23.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:48:23.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:23.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:24.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:25.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:25.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:25.110 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:48:25.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:48:25.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:25.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:25.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:26.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:26.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:26.764 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:48:26.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:48:26.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:26.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:27.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:28.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:28.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:28.193 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:48:28.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:48:28.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:28.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:28.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:29.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:29.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:29.670 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:48:29.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:48:29.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:29.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:30.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:31.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:31.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:31.307 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:48:31.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:48:31.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:31.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:32.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:33.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:33.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:33.195 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:48:33.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:48:33.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:33.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:33.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:34.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:34.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:34.681 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:48:34.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:48:34.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:34.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:35.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:36.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:36.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:36.119 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:48:36.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:48:36.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:36.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:37.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:37.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:37.741 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:48:37.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:48:37.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:37.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:39.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:39.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:39.688 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:48:39.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:48:39.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:39.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:40.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:41.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:41.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:41.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:48:41.441 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:48:41.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:41.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:42.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:43.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:43.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:43.182 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:48:43.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:48:43.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:43.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:44.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:45.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:45.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:45.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:48:45.627 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:48:45.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:45.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:46.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:47.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:47.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:47.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:48:47.558 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:48:47.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:47.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:48.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:48:49.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:48:49.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:49.221 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:48:49.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:48:49.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:49.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:48:49.714 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2708658304,v1:127.0.0.1:6803/2708658304] [v2:127.0.0.1:6804/2708658304,v1:127.0.0.1:6805/2708658304] exists,up 9c49c788-554d-4172-a2c9-0819d6bd8807 2024-04-25T07:48:49.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:48:49.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:48:49.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:48:49.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.0.pid 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=163857 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 1 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/1 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:48:49.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:48:49.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/1' 2024-04-25T07:48:49.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/1/journal' 2024-04-25T07:48:49.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:48:49.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:48:49.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:48:49.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:48:49.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:48:49.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:48:49.718 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:48:49.718 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162711/$cluster-$name.asok' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:48:49.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/1 2024-04-25T07:48:49.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:48:49.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=84a43255-8f33-46ba-ab9a-79dd57c030b2 2024-04-25T07:48:49.720 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 84a43255-8f33-46ba-ab9a-79dd57c030b2 2024-04-25T07:48:49.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 84a43255-8f33-46ba-ab9a-79dd57c030b2' 2024-04-25T07:48:49.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:48:49.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhCipml4ItLBAAzd50PHp92Tj5d+QWXDl+jg== 2024-04-25T07:48:49.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhCipml4ItLBAAzd50PHp92Tj5d+QWXDl+jg=="}' 2024-04-25T07:48:49.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 84a43255-8f33-46ba-ab9a-79dd57c030b2 -i td/osd-fast-mark-down/1/new.json 2024-04-25T07:48:50.325 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:48:50.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/1/new.json 2024-04-25T07:48:50.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhCipml4ItLBAAzd50PHp92Tj5d+QWXDl+jg== --osd-uuid 84a43255-8f33-46ba-ab9a-79dd57c030b2 2024-04-25T07:48:50.378 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:50.381+0000 7fcdb54917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:50.380 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:50.381+0000 7fcdb54917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:50.382 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:50.385+0000 7fcdb54917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:50.382 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:50.385+0000 7fcdb54917c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:48:50.383 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:50.385+0000 7fcdb54917c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:48:50.383 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:50.385+0000 7fcdb54917c0 -1 bluestore(td/osd-fast-mark-down/1/block) _read_bdev_label failed to open td/osd-fast-mark-down/1/block: (2) No such file or directory 2024-04-25T07:48:50.483 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:50.485+0000 7fcdb54917c0 -1 bluestore(td/osd-fast-mark-down/1) _read_fsid unparsable uuid 2024-04-25T07:48:58.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/1/keyring 2024-04-25T07:48:58.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:48:58.523 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:48:58.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:48:58.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:48:59.083 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:48:59.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:48:59.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/1 --osd-journal=td/osd-fast-mark-down/1/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:48:59.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:48:59.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:48:59.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:48:59.116 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:59.117+0000 7f789798c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:59.118 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:59.121+0000 7f789798c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:59.119 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:59.121+0000 7f789798c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:48:59.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:48:59.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:48:59.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:48:59.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:48:59.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:48:59.503 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:48:59.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:48:59.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:48:59.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:48:59.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:48:59.666 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:48:59.669+0000 7f789798c7c0 -1 Falling back to public interface 2024-04-25T07:48:59.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:00.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:00.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:00.923 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:49:00.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:49:00.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:00.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:01.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:02.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:02.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:02.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:49:02.332 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:49:02.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:02.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:02.522 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:02.525+0000 7f789798c7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:49:02.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:49:03.836 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:49:03.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:03.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:04.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:05.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:05.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:05.303 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:49:05.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:49:05.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:05.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:05.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:06.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:06.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:06.791 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:49:06.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:49:06.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:06.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:07.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:08.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:08.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:08.313 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:49:08.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:49:08.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:08.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:08.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:09.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:09.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:09.760 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:49:09.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:49:09.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:09.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:10.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:11.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:11.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:11.382 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:49:11.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:49:11.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:11.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:12.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:13.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:13.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:13.071 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:49:13.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:49:13.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:13.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:13.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:14.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:14.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:14.489 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:49:14.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:49:14.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:14.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:15.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:16.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:16.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:16.144 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:49:16.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:49:16.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:16.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:16.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:17.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:17.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:17.891 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:49:17.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:49:17.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:17.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:18.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:19.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:19.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:19.393 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:49:19.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:49:19.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:19.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:20.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:21.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:21.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:21.046 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:49:21.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:49:21.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:21.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:21.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:22.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:22.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:22.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:49:22.642 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:49:22.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:22.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:23.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:24.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:24.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:24.271 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:49:24.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:49:24.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:24.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:25.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:26.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:26.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:26.211 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:49:26.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:49:26.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:26.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:27.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:28.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:28.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:28.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:49:28.274 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:49:28.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:28.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:29.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:30.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:30.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:30.076 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:49:30.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:49:30.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:30.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:30.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:31.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:31.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:31.865 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:49:31.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:49:31.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:31.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:32.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:33.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:33.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:33.829 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:49:33.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:49:33.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:33.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:34.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:35.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:35.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:35.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:49:35.333 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:49:35.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:35.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:49:35.787 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2918996650,v1:127.0.0.1:6811/2918996650] [v2:127.0.0.1:6812/2918996650,v1:127.0.0.1:6813/2918996650] exists,up 84a43255-8f33-46ba-ab9a-79dd57c030b2 2024-04-25T07:49:35.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:49:35.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:49:35.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:49:35.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.1.pid 2024-04-25T07:49:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=165721 2024-04-25T07:49:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:51: test_fast_kill: for oi in {0..2} 2024-04-25T07:49:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:52: test_fast_kill: run_osd td/osd-fast-mark-down 2 2024-04-25T07:49:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-fast-mark-down 2024-04-25T07:49:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:49:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:49:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:49:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-fast-mark-down/2 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126' 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-fast-mark-down/2' 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-fast-mark-down/2/journal' 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-fast-mark-down' 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:49:35.793 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.162711/$cluster-$name.asok' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-fast-mark-down/$name.log' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-fast-mark-down/$name.pid' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:49:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:49:35.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:49:35.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-fast-mark-down/2 2024-04-25T07:49:35.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:49:35.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a82896d3-348f-47c3-bbdb-b5479c6ba3ac 2024-04-25T07:49:35.795 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 a82896d3-348f-47c3-bbdb-b5479c6ba3ac 2024-04-25T07:49:35.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 a82896d3-348f-47c3-bbdb-b5479c6ba3ac' 2024-04-25T07:49:35.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:49:35.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPCypmcem0MBAA4UPf9Yq0if20rH4KUgCtMA== 2024-04-25T07:49:35.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPCypmcem0MBAA4UPf9Yq0if20rH4KUgCtMA=="}' 2024-04-25T07:49:35.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a82896d3-348f-47c3-bbdb-b5479c6ba3ac -i td/osd-fast-mark-down/2/new.json 2024-04-25T07:49:36.324 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:49:36.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-fast-mark-down/2/new.json 2024-04-25T07:49:36.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPCypmcem0MBAA4UPf9Yq0if20rH4KUgCtMA== --osd-uuid a82896d3-348f-47c3-bbdb-b5479c6ba3ac 2024-04-25T07:49:36.377 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:36.376+0000 7f43cd4cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:36.379 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:36.380+0000 7f43cd4cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:36.380 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:36.380+0000 7f43cd4cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:36.381 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:36.380+0000 7f43cd4cc7c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:49:36.381 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:36.380+0000 7f43cd4cc7c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:49:36.381 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:36.380+0000 7f43cd4cc7c0 -1 bluestore(td/osd-fast-mark-down/2/block) _read_bdev_label failed to open td/osd-fast-mark-down/2/block: (2) No such file or directory 2024-04-25T07:49:36.450 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:36.452+0000 7f43cd4cc7c0 -1 bluestore(td/osd-fast-mark-down/2) _read_fsid unparsable uuid 2024-04-25T07:49:44.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-fast-mark-down/2/keyring 2024-04-25T07:49:44.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:49:44.492 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:49:44.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:49:44.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-fast-mark-down/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:49:45.071 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:49:45.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:49:45.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=d80a2d6f-f2fc-4b4d-9de8-2cd733b76e3a --auth-supported=none --osd-fast-fail-on-connection-refused=true --ms_type=async --mon-host=127.0.0.1:7126 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-fast-mark-down/2 --osd-journal=td/osd-fast-mark-down/2/journal --chdir= --run-dir=td/osd-fast-mark-down '--admin-socket=/tmp/ceph-asok.162711/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-fast-mark-down/$name.log' '--pid-file=td/osd-fast-mark-down/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:49:45.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:49:45.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:49:45.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:49:45.104 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:45.104+0000 7f1fe2de37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:45.107 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:45.108+0000 7f1fe2de37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:45.108 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:45.108+0000 7f1fe2de37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:49:45.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:49:45.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:49:45.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:49:45.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:49:45.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:49:45.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:45.476 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:49:45.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:49:45.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:45.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:45.683 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:45.684+0000 7f1fe2de37c0 -1 Falling back to public interface 2024-04-25T07:49:45.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:46.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:46.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:46.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:49:46.908 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:49:46.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:46.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:47.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:48.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:48.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:48.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:49:48.410 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:49:48.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:48.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:48.472 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:49:48.472+0000 7f1fe2de37c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:49:48.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:49.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:49.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:49.946 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:49:49.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:49:49.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:49.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:50.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:51.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:51.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:51.385 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:49:51.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:49:51.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:51.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:51.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:52.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:52.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:52.889 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:49:52.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:49:52.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:52.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:53.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:54.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:54.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:54.384 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:49:54.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:49:54.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:54.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:54.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:55.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:55.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:55.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:49:55.869 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:49:55.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:55.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:56.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:57.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:57.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:57.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:49:57.337 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:49:57.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:57.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:57.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:49:58.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:49:58.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:49:58.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:49:58.837 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:49:58.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:49:58.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:49:59.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:00.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:00.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:00.339 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:50:00.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:50:00.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:00.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:00.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:01.829 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:50:01.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:50:01.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:01.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:02.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:03.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:03.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:03.383 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:50:03.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:50:03.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:03.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:03.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:04.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:04.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:04.981 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:50:04.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:50:04.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:04.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:05.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:06.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:06.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:06.687 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:50:06.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:50:06.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:06.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:07.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:08.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:08.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:08.199 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:50:08.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:50:08.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:08.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:09.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:10.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:10.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:10.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:50:10.390 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:50:10.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:10.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:11.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:12.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:12.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:12.417 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:50:12.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:50:12.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:12.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:13.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:14.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:14.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:14.865 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:50:14.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:50:14.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:14.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:15.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:16.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:16.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:16.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:50:16.513 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:50:16.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:16.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:17.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:18.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:18.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:18.338 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:50:18.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:50:18.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:18.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:19.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:20.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:20.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:20.460 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:50:20.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:50:20.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:20.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:21.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:22.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:22.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:22.028 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:50:22.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:50:22.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:22.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:50:22.441 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1195493617,v1:127.0.0.1:6819/1195493617] [v2:127.0.0.1:6820/1195493617,v1:127.0.0.1:6821/1195493617] exists,up a82896d3-348f-47c3-bbdb-b5479c6ba3ac 2024-04-25T07:50:22.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:50:22.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:50:22.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:50:22.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: cat td/osd-fast-mark-down/osd.2.pid 2024-04-25T07:50:22.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:53: test_fast_kill: pids[$oi]=167585 2024-04-25T07:50:22.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:56: test_fast_kill: create_rbd_pool 2024-04-25T07:50:22.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:50:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' does not exist 2024-04-25T07:50:22.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:50:22.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:50:23.694 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' already exists 2024-04-25T07:50:23.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:50:24.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:50:27.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:59: test_fast_kill: timeout 20 rados -p rbd bench 10 write -b 4096 --max-objects 128 --no-cleanup 2024-04-25T07:50:27.837 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T07:50:27.837 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 128 objects 2024-04-25T07:50:27.837 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_168683 2024-04-25T07:50:27.837 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T07:50:27.838 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T07:50:28.838 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 64 48 0.187479 0.1875 0.368678 0.301139 2024-04-25T07:50:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: 2 16 112 96 0.18748 0.1875 0.327545 0.327922 2024-04-25T07:50:30.838 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 2.71906 2024-04-25T07:50:30.838 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 128 2024-04-25T07:50:30.838 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 4096 2024-04-25T07:50:30.838 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 4096 2024-04-25T07:50:30.838 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 0.183887 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 0 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 0.1875 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 0.1875 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 47 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 0 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 48 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 48 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 0.338228 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.0516941 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 0.405721 2024-04-25T07:50:30.839 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.187479 2024-04-25T07:50:31.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:60: test_fast_kill: sleep 1 2024-04-25T07:50:32.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:62: test_fast_kill: killid=0 2024-04-25T07:50:32.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:63: test_fast_kill: previd=0 2024-04-25T07:50:32.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-04-25T07:50:32.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 0 -eq 0 ']' 2024-04-25T07:50:32.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=167585 2024-04-25T07:50:32.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 167585 -eq 0 ']' 2024-04-25T07:50:32.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=167585 2024-04-25T07:50:32.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 167585 2024-04-25T07:50:32.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-04-25T07:50:32.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-04-25T07:50:32.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-04-25T07:50:32.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-04-25T07:50:33.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-04-25T07:50:33.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-04-25T07:50:33.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-04-25T07:50:33.023 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-04-25T07:50:33.023 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-04-25T07:50:33.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=1 2024-04-25T07:50:33.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 1 -lt 1 ']' 2024-04-25T07:50:33.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 1 -gt 1 ']' 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 1 -lt 1 ']' 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:66: test_fast_kill: for i in {1..2} 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 167585 -eq 167585 ']' 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:68: test_fast_kill: killid=163857 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:67: test_fast_kill: '[' 163857 -eq 167585 ']' 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:70: test_fast_kill: previd=163857 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:72: test_fast_kill: kill -9 163857 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:73: test_fast_kill: time_left=30 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:74: test_fast_kill: down_osds=0 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:76: test_fast_kill: '[' 30 -gt 0 ']' 2024-04-25T07:50:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:77: test_fast_kill: sleep 1 2024-04-25T07:50:34.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:78: test_fast_kill: time_left=29 2024-04-25T07:50:34.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:80: test_fast_kill: grep -m 1 -c -F ms_handle_refused td/osd-fast-mark-down/osd.0.log td/osd-fast-mark-down/osd.1.log td/osd-fast-mark-down/osd.2.log 2024-04-25T07:50:34.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:81: test_fast_kill: '[' 0 -ne 0 ']' 2024-04-25T07:50:34.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: ceph osd tree 2024-04-25T07:50:34.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: grep -c down 2024-04-25T07:50:35.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:85: test_fast_kill: down_osds=2 2024-04-25T07:50:35.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:86: test_fast_kill: '[' 2 -lt 2 ']' 2024-04-25T07:50:35.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:89: test_fast_kill: '[' 2 -gt 2 ']' 2024-04-25T07:50:35.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:93: test_fast_kill: break 2024-04-25T07:50:35.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:97: test_fast_kill: '[' 2 -lt 2 ']' 2024-04-25T07:50:35.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:103: test_fast_kill: pkill -SIGTERM rados 2024-04-25T07:50:35.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:104: test_fast_kill: teardown td/osd-fast-mark-down 2024-04-25T07:50:35.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:50:35.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:50:35.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:50:35.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:50:35.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:50:35.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:50:35.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:50:35.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:50:35.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:50:35.463 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:50:35.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:50:35.463 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:50:35.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:50:35.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:50:35.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:50:35.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:35.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:50:35.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:50:35.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:35.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:50:35.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:50:35.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:50:35.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:50:35.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:35.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:50:35.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162711 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-fast-mark-down.sh:42: run: return 0 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-fast-mark-down 0 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-fast-mark-down 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-fast-mark-down KILL 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:50:35.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:50:35.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:50:35.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:50:35.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:50:35.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:50:35.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:50:35.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:50:35.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:50:35.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:50:35.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:50:35.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:35.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:50:35.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:50:35.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:35.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:50:35.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:50:35.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-fast-mark-down 2024-04-25T07:50:35.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:50:35.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:35.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.162711 2024-04-25T07:50:35.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.162711 2024-04-25T07:50:35.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:50:35.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:50:35.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:50:35.494 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:50:35.495 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:50:35.546 INFO:tasks.workunit:Running workunit osd/osd-force-create-pg.sh... 2024-04-25T07:50:35.546 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-force-create-pg.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh 2024-04-25T07:50:35.597 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:50:35.603 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:50:35.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:50:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:50:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:50:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:50:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:50:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:50:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:50:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:50:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-force-create-pg 2024-04-25T07:50:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:5: run: local dir=td/osd-force-create-pg 2024-04-25T07:50:35.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:6: run: shift 2024-04-25T07:50:35.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: export CEPH_MON=127.0.0.1:7145 2024-04-25T07:50:35.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:8: run: CEPH_MON=127.0.0.1:7145 2024-04-25T07:50:35.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:9: run: export CEPH_ARGS 2024-04-25T07:50:35.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: uuidgen 2024-04-25T07:50:35.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:10: run: CEPH_ARGS+='--fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none ' 2024-04-25T07:50:35.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7145 ' 2024-04-25T07:50:35.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: set 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:13: run: local funcs=TEST_reuse_id 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:14: run: for func in $funcs 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:15: run: setup td/osd-force-create-pg 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-force-create-pg 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-force-create-pg 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:50:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:50:35.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:50:35.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:50:35.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:50:35.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:50:35.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:50:35.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:50:35.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:50:35.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:35.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:50:35.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:50:35.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:50:35.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:50:35.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:50:35.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:50:35.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:50:35.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:35.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:50:35.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168814 2024-04-25T07:50:35.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:50:35.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:50:35.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-force-create-pg 2024-04-25T07:50:35.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:50:35.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:35.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:50:35.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.168814 2024-04-25T07:50:35.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-force-create-pg 1' TERM HUP INT 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:16: run: TEST_reuse_id td/osd-force-create-pg 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:22: TEST_reuse_id: local dir=td/osd-force-create-pg 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:24: TEST_reuse_id: run_mon td/osd-force-create-pg a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-force-create-pg 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-force-create-pg/a 2024-04-25T07:50:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-force-create-pg/a --run-dir=td/osd-force-create-pg --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:50:36.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:50:36.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:50:36.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:50:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:50:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:50:36.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:50:36.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-force-create-pg/a '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --mon-cluster-log-file=td/osd-force-create-pg/log --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T07:50:36.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:50:36.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:50:36.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:50:36.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:50:36.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:50:36.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:50:36.355 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:50:36.356 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:50:36.356 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:50:36.356 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:50:36.356 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:36.356 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:50:36.356 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168814/ceph-mon.a.asok 2024-04-25T07:50:36.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:50:36.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168814/ceph-mon.a.asok config get fsid 2024-04-25T07:50:36.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:50:36.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:50:36.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:50:36.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:50:36.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:50:36.471 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:50:36.471 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:50:36.471 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:50:36.471 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:50:36.471 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:36.471 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:50:36.471 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.168814/ceph-mon.a.asok 2024-04-25T07:50:36.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:50:36.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.168814/ceph-mon.a.asok config get mon_host 2024-04-25T07:50:36.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:25: TEST_reuse_id: run_mgr td/osd-force-create-pg x 2024-04-25T07:50:36.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-force-create-pg 2024-04-25T07:50:36.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:50:36.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:50:36.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:50:36.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-force-create-pg/x 2024-04-25T07:50:36.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:50:37.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:50:37.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:50:37.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:50:37.545 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:50:37.545 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:37.545 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:50:37.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:50:37.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:50:37.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-force-create-pg/x '--log-file=td/osd-force-create-pg/$name.log' '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --run-dir=td/osd-force-create-pg '--pid-file=td/osd-force-create-pg/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:50:37.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:26: TEST_reuse_id: run_osd td/osd-force-create-pg 0 2024-04-25T07:50:37.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:50:37.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:50:37.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:50:37.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/0 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:50:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:50:37.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/0 2024-04-25T07:50:37.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:50:37.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=01a3fd86-9473-4863-ba69-6b5052573588 2024-04-25T07:50:37.585 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 01a3fd86-9473-4863-ba69-6b5052573588 2024-04-25T07:50:37.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 01a3fd86-9473-4863-ba69-6b5052573588' 2024-04-25T07:50:37.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:50:37.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBNCypm0f32IxAAHSI5RWQO3nBOWQ1nY643sQ== 2024-04-25T07:50:37.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBNCypm0f32IxAAHSI5RWQO3nBOWQ1nY643sQ=="}' 2024-04-25T07:50:37.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 01a3fd86-9473-4863-ba69-6b5052573588 -i td/osd-force-create-pg/0/new.json 2024-04-25T07:50:37.948 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:50:37.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/0/new.json 2024-04-25T07:50:37.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBNCypm0f32IxAAHSI5RWQO3nBOWQ1nY643sQ== --osd-uuid 01a3fd86-9473-4863-ba69-6b5052573588 2024-04-25T07:50:37.989 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:37.990+0000 7f292ffeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:37.991 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:37.990+0000 7f292ffeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:37.993 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:37.990+0000 7f292ffeb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:37.993 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:37.994+0000 7f292ffeb7c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:50:37.993 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:37.994+0000 7f292ffeb7c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:50:37.993 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:37.994+0000 7f292ffeb7c0 -1 bluestore(td/osd-force-create-pg/0/block) _read_bdev_label failed to open td/osd-force-create-pg/0/block: (2) No such file or directory 2024-04-25T07:50:38.082 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:38.082+0000 7f292ffeb7c0 -1 bluestore(td/osd-force-create-pg/0) _read_fsid unparsable uuid 2024-04-25T07:50:46.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/0/keyring 2024-04-25T07:50:46.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:50:46.037 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:50:46.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:50:46.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:50:46.617 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:50:46.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:50:46.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:50:46.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:50:46.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:50:46.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:50:46.648 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:46.646+0000 7f87f5a1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:46.650 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:46.650+0000 7f87f5a1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:46.652 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:46.650+0000 7f87f5a1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:50:47.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:50:47.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:50:47.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:50:47.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:50:47.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:50:47.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:47.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:50:47.010 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:50:47.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:47.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:50:47.200 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:47.198+0000 7f87f5a1a7c0 -1 Falling back to public interface 2024-04-25T07:50:47.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:48.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:48.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:48.431 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:50:48.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:50:48.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:48.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:50:48.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:49.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:49.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:49.845 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:50:49.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:50:49.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:49.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:50:50.060 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:50:50.058+0000 7f87f5a1a7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:50:50.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:51.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:51.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:51.324 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:50:51.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:50:51.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:51.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:50:51.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:52.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:52.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:52.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:50:52.795 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:50:52.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:52.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:50:53.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:54.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:54.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:54.240 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:50:54.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:50:54.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:54.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:50:54.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:55.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:55.694 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:50:55.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:55.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:50:55.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:55.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:50:56.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:57.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:57.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:57.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:50:57.278 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:50:57.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:57.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:50:57.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:50:58.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:50:58.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:50:58.728 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:50:58.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:50:58.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:50:58.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:50:59.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:00.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:00.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:00.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:51:00.277 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:51:00.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:00.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:00.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:01.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:01.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:01.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:51:01.865 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:51:01.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:01.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:03.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:03.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:03.399 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:51:03.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:51:03.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:03.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:03.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:04.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:04.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:04.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:51:04.899 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:51:04.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:04.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:05.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:06.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:06.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:06.419 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:51:06.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:51:06.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:06.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:06.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:07.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:07.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:07.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:51:07.971 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:51:07.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:07.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:08.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:09.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:09.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:09.487 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:51:09.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:51:09.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:09.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:10.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:11.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:11.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:11.037 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:51:11.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:51:11.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:11.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:12.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:13.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:13.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:13.370 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:51:13.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:51:13.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:13.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:14.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:15.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:15.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:15.306 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:51:15.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:51:15.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:15.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:16.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:17.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:17.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:17.256 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:51:17.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:51:17.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:17.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:18.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:19.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:19.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:19.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:51:19.183 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:51:19.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:19.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:19.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:20.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:20.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:20.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:51:20.938 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:51:20.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:20.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:21.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:22.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:22.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:22.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:51:22.473 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:51:22.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:22.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:22.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:23.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:23.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:23.932 INFO:tasks.workunit.client.0.smithi134.stdout:23 2024-04-25T07:51:23.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-04-25T07:51:23.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:23.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:51:24.351 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3400549157,v1:127.0.0.1:6803/3400549157] [v2:127.0.0.1:6804/3400549157,v1:127.0.0.1:6805/3400549157] exists,up 01a3fd86-9473-4863-ba69-6b5052573588 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:27: TEST_reuse_id: run_osd td/osd-force-create-pg 1 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/1 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:51:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:51:24.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:51:24.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:51:24.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/1 2024-04-25T07:51:24.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:51:24.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=53a94ad6-6398-485e-a660-c7d220d5b2c7 2024-04-25T07:51:24.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 53a94ad6-6398-485e-a660-c7d220d5b2c7' 2024-04-25T07:51:24.359 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 53a94ad6-6398-485e-a660-c7d220d5b2c7 2024-04-25T07:51:24.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:51:24.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB8CypmL1NhFhAAQVaYQtCKAdW+KxBy4tzxDg== 2024-04-25T07:51:24.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB8CypmL1NhFhAAQVaYQtCKAdW+KxBy4tzxDg=="}' 2024-04-25T07:51:24.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 53a94ad6-6398-485e-a660-c7d220d5b2c7 -i td/osd-force-create-pg/1/new.json 2024-04-25T07:51:25.013 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:51:25.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/1/new.json 2024-04-25T07:51:25.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB8CypmL1NhFhAAQVaYQtCKAdW+KxBy4tzxDg== --osd-uuid 53a94ad6-6398-485e-a660-c7d220d5b2c7 2024-04-25T07:51:25.062 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:25.065+0000 7f0dc3d8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:25.064 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:25.065+0000 7f0dc3d8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:25.065 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:25.069+0000 7f0dc3d8e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:25.066 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:25.069+0000 7f0dc3d8e7c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:51:25.066 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:25.069+0000 7f0dc3d8e7c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:51:25.066 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:25.069+0000 7f0dc3d8e7c0 -1 bluestore(td/osd-force-create-pg/1/block) _read_bdev_label failed to open td/osd-force-create-pg/1/block: (2) No such file or directory 2024-04-25T07:51:25.130 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:25.133+0000 7f0dc3d8e7c0 -1 bluestore(td/osd-force-create-pg/1) _read_fsid unparsable uuid 2024-04-25T07:51:33.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/1/keyring 2024-04-25T07:51:33.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:51:33.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:51:33.184 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:51:33.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:51:33.762 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:51:33.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:51:33.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:51:33.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:51:33.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:51:33.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:51:33.791 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:33.793+0000 7f014b2b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:33.793 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:33.793+0000 7f014b2b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:33.794 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:33.797+0000 7f014b2b17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:51:34.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:51:34.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:51:34.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:51:34.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:51:34.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:51:34.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:34.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:51:34.168 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:51:34.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:34.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:34.347 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:34.349+0000 7f014b2b17c0 -1 Falling back to public interface 2024-04-25T07:51:34.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:35.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:35.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:35.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:51:35.586 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:51:35.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:35.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:35.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:36.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:36.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:36.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:51:36.988 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:51:36.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:36.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:37.148 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:51:37.149+0000 7f014b2b17c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:51:37.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:38.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:38.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:38.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:51:38.473 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:51:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:38.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:39.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:40.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:40.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:40.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:51:40.029 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:51:40.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:40.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:40.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:41.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:41.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:41.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:51:41.891 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:51:41.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:41.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:42.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:43.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:43.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:43.732 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:51:43.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:51:43.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:43.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:44.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:45.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:45.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:45.475 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:51:45.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:51:45.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:45.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:45.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:46.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:46.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:46.931 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:51:46.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:51:46.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:46.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:47.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:48.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:48.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:48.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:51:48.486 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:51:48.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:48.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:49.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:50.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:50.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:50.073 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:51:50.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:51:50.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:50.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:50.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:51.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:51.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:51.773 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:51:51.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:51:51.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:51.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:52.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:53.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:53.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:53.216 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:51:53.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:51:53.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:53.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:53.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:54.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:54.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:54.819 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:51:54.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:51:54.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:54.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:55.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:56.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:56.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:56.462 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:51:56.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:51:56.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:56.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:57.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:51:58.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:51:58.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:51:58.324 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:51:58.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:51:58.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:51:58.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:51:59.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:00.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:00.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:00.915 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:52:00.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:52:00.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:00.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:01.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:02.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:02.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:02.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:52:02.859 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:52:02.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:02.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:04.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:05.256 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:52:05.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:52:05.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:05.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:06.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:07.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:07.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:07.066 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:52:07.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:52:07.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:07.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:07.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:08.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:08.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:08.916 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:52:08.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:52:08.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:08.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:09.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:10.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:10.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:10.881 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:52:10.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:52:10.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:10.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:52:11.295 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2046003551,v1:127.0.0.1:6811/2046003551] [v2:127.0.0.1:6812/2046003551,v1:127.0.0.1:6813/2046003551] exists,up 53a94ad6-6398-485e-a660-c7d220d5b2c7 2024-04-25T07:52:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:52:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:52:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:52:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:28: TEST_reuse_id: run_osd td/osd-force-create-pg 2 2024-04-25T07:52:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-force-create-pg 2024-04-25T07:52:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:52:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:52:11.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-force-create-pg/2 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:52:11.299 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:52:11.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:52:11.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:52:11.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-force-create-pg/2 2024-04-25T07:52:11.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:52:11.301 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 ea6aeaed-5927-4630-b8ba-c9803e5e4c34 2024-04-25T07:52:11.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ea6aeaed-5927-4630-b8ba-c9803e5e4c34 2024-04-25T07:52:11.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 ea6aeaed-5927-4630-b8ba-c9803e5e4c34' 2024-04-25T07:52:11.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:52:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrCypmZ7JNExAAE+230opfOJ5porxMCZ+l0g== 2024-04-25T07:52:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrCypmZ7JNExAAE+230opfOJ5porxMCZ+l0g=="}' 2024-04-25T07:52:11.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ea6aeaed-5927-4630-b8ba-c9803e5e4c34 -i td/osd-force-create-pg/2/new.json 2024-04-25T07:52:11.995 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:52:12.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-force-create-pg/2/new.json 2024-04-25T07:52:12.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrCypmZ7JNExAAE+230opfOJ5porxMCZ+l0g== --osd-uuid ea6aeaed-5927-4630-b8ba-c9803e5e4c34 2024-04-25T07:52:12.049 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:12.048+0000 7fb06ee107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:12.050 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:12.052+0000 7fb06ee107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:12.051 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:12.052+0000 7fb06ee107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:12.052 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:12.052+0000 7fb06ee107c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:52:12.052 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:12.052+0000 7fb06ee107c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:52:12.052 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:12.052+0000 7fb06ee107c0 -1 bluestore(td/osd-force-create-pg/2/block) _read_bdev_label failed to open td/osd-force-create-pg/2/block: (2) No such file or directory 2024-04-25T07:52:12.111 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:12.112+0000 7fb06ee107c0 -1 bluestore(td/osd-force-create-pg/2) _read_fsid unparsable uuid 2024-04-25T07:52:19.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-force-create-pg/2/keyring 2024-04-25T07:52:19.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:52:19.105 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:52:19.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:52:19.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-force-create-pg/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:52:19.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:52:19.678 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:52:19.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:52:19.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:52:19.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:52:19.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:52:19.708 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:19.708+0000 7fd27a5457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:19.711 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:19.712+0000 7fd27a5457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:19.712 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:19.712+0000 7fd27a5457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:52:20.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:52:20.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:52:20.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:52:20.083 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:52:20.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:52:20.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:52:20.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:20.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:52:20.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:20.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:20.271 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:20.272+0000 7fd27a5457c0 -1 Falling back to public interface 2024-04-25T07:52:20.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:21.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:21.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:21.554 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:52:21.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:52:21.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:21.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:21.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:22.948 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:52:22.948+0000 7fd27a5457c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:52:22.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:22.958 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:52:22.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:22.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:52:22.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:22.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:23.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:24.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:24.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:24.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:52:24.385 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:52:24.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:24.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:24.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:25.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:25.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:25.958 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:52:25.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:52:25.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:25.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:26.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:27.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:27.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:27.391 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:52:27.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:52:27.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:27.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:27.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:28.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:28.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:28.914 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:52:28.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:52:28.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:28.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:29.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:30.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:30.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:30.454 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:52:30.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:52:30.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:30.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:31.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:32.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:32.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:32.044 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:52:32.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:52:32.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:32.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:32.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:33.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:33.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:33.494 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:52:33.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:52:33.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:33.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:33.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:34.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:34.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:34.985 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:52:34.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:52:34.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:34.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:36.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:36.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:36.446 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:52:36.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:52:36.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:36.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:36.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:37.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:37.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:37.986 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:52:37.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:52:37.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:37.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:38.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:39.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:39.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:39.949 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:52:39.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:52:39.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:39.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:40.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:41.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:41.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:41.996 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:52:41.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:52:41.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:41.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:43.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:44.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:44.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:44.263 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:52:44.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:52:44.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:44.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:45.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:46.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:46.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:46.370 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:52:46.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:52:46.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:46.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:47.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:48.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:48.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:48.726 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:52:48.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:52:48.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:48.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:49.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:50.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:50.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:50.382 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:52:50.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:52:50.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:50.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:50.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:52:51.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:52:51.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:52:51.852 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:52:51.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:52:51.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:52:51.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:52:52.287 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3307988433,v1:127.0.0.1:6819/3307988433] [v2:127.0.0.1:6820/3307988433,v1:127.0.0.1:6821/3307988433] exists,up ea6aeaed-5927-4630-b8ba-c9803e5e4c34 2024-04-25T07:52:52.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:52:52.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:52:52.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:52:52.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:30: TEST_reuse_id: ceph osd pool create foo 50 2024-04-25T07:52:52.989 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T07:52:53.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:31: TEST_reuse_id: wait_for_clean 2024-04-25T07:52:53.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:52:53.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:52:53.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:52:53.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:52:53.007 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:52:53.007 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:52:53.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:52:53.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:52:53.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:52:53.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:52:53.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:52:53.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:52:53.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:52:53.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:52:53.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:52:53.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:52:53.728 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:52:53.728 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:52:53.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:52:53.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:53.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:52:53.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836504 2024-04-25T07:52:53.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836504 2024-04-25T07:52:53.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504' 2024-04-25T07:52:53.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:53.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:52:54.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T07:52:54.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T07:52:54.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978' 2024-04-25T07:52:54.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:52:54.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:52:54.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509444 2024-04-25T07:52:54.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509444 2024-04-25T07:52:54.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836504 1-42949672978 2-64424509444' 2024-04-25T07:52:54.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:54.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836504 2024-04-25T07:52:54.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:54.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:52:54.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836504 2024-04-25T07:52:54.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:54.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836504 2024-04-25T07:52:54.755 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836504 2024-04-25T07:52:54.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-04-25T07:52:54.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:52:55.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836504 -lt 21474836504 2024-04-25T07:52:55.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:55.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T07:52:55.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:55.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:52:55.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T07:52:55.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:55.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T07:52:55.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T07:52:55.421 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672978 2024-04-25T07:52:55.421 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:52:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672976 -lt 42949672978 2024-04-25T07:52:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:52:57.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:52:57.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:52:57.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-04-25T07:52:57.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:52:57.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509444 2024-04-25T07:52:57.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:52:57.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:52:57.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509444 2024-04-25T07:52:57.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:52:57.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509444 2024-04-25T07:52:57.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2024-04-25T07:52:57.612 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509444 2024-04-25T07:52:57.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:52:58.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509444 -lt 64424509444 2024-04-25T07:52:58.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:52:58.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:52:58.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:52:58.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-04-25T07:52:58.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:52:58.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:52:58.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:52:58.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:52:58.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:52:58.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:52:58.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:52:59.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=36 2024-04-25T07:52:59.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:52:59.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:52:59.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:52:59.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 36 = 50 2024-04-25T07:52:59.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 36 '!=' -1 2024-04-25T07:52:59.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:52:59.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=36 2024-04-25T07:52:59.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:52:59.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:52:59.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:52:59.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:52:59.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:52:59.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:52:59.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:52:59.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:52:59.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:52:59.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:53:00.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-04-25T07:53:00.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:53:00.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:53:00.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:53:00.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-04-25T07:53:00.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:53:00.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:53:00.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:33: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.0 2024-04-25T07:53:00.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:53:00.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:53:00.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:53:00.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:53:00.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:53:01.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:53:01.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:34: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.1 2024-04-25T07:53:01.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:53:01.035 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:53:01.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:53:01.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:53:01.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:53:01.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:53:01.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:35: TEST_reuse_id: kill_daemons td/osd-force-create-pg TERM osd.2 2024-04-25T07:53:01.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:53:01.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:53:01.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:53:01.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:53:01.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:53:01.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:53:01.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:36: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/0 --op remove --pgid 1.0 --force 2024-04-25T07:53:03.421 INFO:tasks.workunit.client.0.smithi134.stderr:PG '1.0' not found 2024-04-25T07:53:03.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:37: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/1 --op remove --pgid 1.0 --force 2024-04-25T07:53:05.484 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T07:53:05.484 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T07:53:05.484 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T07:53:05.633 INFO:tasks.workunit.client.0.smithi134.stdout:Remove successful 2024-04-25T07:53:06.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:38: TEST_reuse_id: ceph-objectstore-tool --data-path td/osd-force-create-pg/2 --op remove --pgid 1.0 --force 2024-04-25T07:53:07.331 INFO:tasks.workunit.client.0.smithi134.stderr:PG '1.0' not found 2024-04-25T07:53:07.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:39: TEST_reuse_id: activate_osd td/osd-force-create-pg 0 2024-04-25T07:53:07.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/0 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/0' 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/0/journal' 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:53:07.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:53:07.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:53:07.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:53:07.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:53:07.867 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:53:07.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/0 2024-04-25T07:53:07.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T07:53:07.869 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:53:07.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/0 --osd-journal=td/osd-force-create-pg/0/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:53:07.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/0/whoami 2024-04-25T07:53:07.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T07:53:07.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:53:07.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:53:07.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:53:07.905 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:07.902+0000 7f1b5c7977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:07.908 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:07.906+0000 7f1b5c7977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:07.909 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:07.906+0000 7f1b5c7977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:08.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T07:53:08.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:53:08.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:53:08.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:53:08.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:53:08.333 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:53:08.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:08.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:53:08.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:08.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:08.463 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:08.462+0000 7f1b5c7977c0 -1 Falling back to public interface 2024-04-25T07:53:08.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:09.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:09.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:09.797 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:53:09.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:53:09.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:09.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:10.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:11.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:11.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:11.211 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:53:11.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:53:11.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:11.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:11.362 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:11.362+0000 7f1b5c7977c0 -1 osd.0 24 log_to_monitors true 2024-04-25T07:53:11.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:12.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:12.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:12.632 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:53:12.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:53:12.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:12.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:13.168 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 29 up_thru 29 down_at 25 last_clean_interval [5,24) [v2:127.0.0.1:6802/3575451947,v1:127.0.0.1:6803/3575451947] [v2:127.0.0.1:6804/3575451947,v1:127.0.0.1:6805/3575451947] exists,up 01a3fd86-9473-4863-ba69-6b5052573588 2024-04-25T07:53:13.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:53:13.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:53:13.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:53:13.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:40: TEST_reuse_id: activate_osd td/osd-force-create-pg 1 2024-04-25T07:53:13.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:53:13.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:53:13.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T07:53:13.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:53:13.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/1 2024-04-25T07:53:13.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:53:13.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:53:13.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:53:13.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:53:13.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/1' 2024-04-25T07:53:13.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/1/journal' 2024-04-25T07:53:13.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:53:13.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:53:13.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:53:13.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:53:13.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:53:13.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:53:13.176 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:53:13.176 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:13.176 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:53:13.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:53:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:53:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:53:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:53:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:53:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:53:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:53:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:53:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:53:13.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:53:13.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:53:13.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/1 2024-04-25T07:53:13.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T07:53:13.178 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:53:13.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/1 --osd-journal=td/osd-force-create-pg/1/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:53:13.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/1/whoami 2024-04-25T07:53:13.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T07:53:13.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:53:13.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:53:13.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:53:13.201 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:13.198+0000 7f87a8f4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:13.244 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:13.242+0000 7f87a8f4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:13.245 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:13.242+0000 7f87a8f4e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:13.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T07:53:13.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:53:13.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:53:13.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:53:13.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:53:13.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:13.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:53:13.577 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:53:13.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:13.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:13.780 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:13.778+0000 7f87a8f4e7c0 -1 Falling back to public interface 2024-04-25T07:53:14.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:15.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:15.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:15.041 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:53:15.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:53:15.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:15.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:15.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:16.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:16.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:16.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:53:16.449 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:53:16.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:16.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:16.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:16.941 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:16.938+0000 7f87a8f4e7c0 -1 osd.1 25 log_to_monitors true 2024-04-25T07:53:17.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:17.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:17.938 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:53:17.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:53:17.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:17.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:53:18.560 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 32 up_thru 32 down_at 26 last_clean_interval [10,25) [v2:127.0.0.1:6810/759894444,v1:127.0.0.1:6811/759894444] [v2:127.0.0.1:6812/759894444,v1:127.0.0.1:6813/759894444] exists,up 53a94ad6-6398-485e-a660-c7d220d5b2c7 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:41: TEST_reuse_id: activate_osd td/osd-force-create-pg 2 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-force-create-pg 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-force-create-pg/2 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 ' 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:53:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-force-create-pg/2' 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-force-create-pg/2/journal' 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-force-create-pg' 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:53:18.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:53:18.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-force-create-pg/$name.log' 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-force-create-pg/$name.pid' 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-force-create-pg/2 2024-04-25T07:53:18.565 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:53:18.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T07:53:18.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=2deddfdd-d89e-4e61-b9c7-7795e7a7f427 --auth-supported=none --mon-host=127.0.0.1:7145 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-force-create-pg/2 --osd-journal=td/osd-force-create-pg/2/journal --chdir= --run-dir=td/osd-force-create-pg '--admin-socket=/tmp/ceph-asok.168814/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-force-create-pg/$name.log' '--pid-file=td/osd-force-create-pg/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:53:18.566 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-force-create-pg/2/whoami 2024-04-25T07:53:18.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T07:53:18.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T07:53:18.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T07:53:18.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T07:53:18.593 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:18.590+0000 7f65c353f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:18.596 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:18.594+0000 7f65c353f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:18.598 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:18.598+0000 7f65c353f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:18.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T07:53:18.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:53:18.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:53:18.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:53:18.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:53:18.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:18.989 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:53:18.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:53:18.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:18.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:19.155 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:19.154+0000 7f65c353f7c0 -1 Falling back to public interface 2024-04-25T07:53:19.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:20.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:20.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:20.402 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:53:20.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:53:20.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:20.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:20.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:21.678 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:21.674+0000 7f65c353f7c0 -1 osd.2 26 log_to_monitors true 2024-04-25T07:53:21.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:21.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:21.831 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:53:21.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:53:21.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:21.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:53:22.440 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 35 up_thru 17 down_at 27 last_clean_interval [15,26) [v2:127.0.0.1:6818/454873957,v1:127.0.0.1:6819/454873957] [v2:127.0.0.1:6820/454873957,v1:127.0.0.1:6821/454873957] exists,up ea6aeaed-5927-4630-b8ba-c9803e5e4c34 2024-04-25T07:53:22.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:53:22.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:53:22.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:53:22.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:42: TEST_reuse_id: sleep 10 2024-04-25T07:53:32.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: ceph pg ls 2024-04-25T07:53:32.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep 1.0 2024-04-25T07:53:32.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:43: TEST_reuse_id: grep stale 2024-04-25T07:53:32.864 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 stale+active+clean 38s 0'0 24:20 [1]p1 [1]p1 2024-04-25T07:52:52.692811+0000 2024-04-25T07:52:52.692811+0000 0 periodic scrub scheduled @ 2024-04-26T10:17:05.304321+0000 2024-04-25T07:53:32.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:45: TEST_reuse_id: ceph osd force-create-pg 1.0 --yes-i-really-mean-it 2024-04-25T07:53:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:pg 1.0 already creating 2024-04-25T07:53:33.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:46: TEST_reuse_id: wait_for_clean 2024-04-25T07:53:33.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T07:53:33.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T07:53:33.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T07:53:33.413 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T07:53:33.413 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T07:53:33.413 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T07:53:33.414 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T07:53:33.414 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T07:53:33.414 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T07:53:33.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T07:53:33.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T07:53:33.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T07:53:33.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T07:53:33.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T07:53:33.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T07:53:33.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T07:53:33.904 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T07:53:33.904 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T07:53:33.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T07:53:33.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:53:33.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T07:53:34.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051591 2024-04-25T07:53:34.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051591 2024-04-25T07:53:34.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051591' 2024-04-25T07:53:34.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:53:34.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T07:53:34.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953478 2024-04-25T07:53:34.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953478 2024-04-25T07:53:34.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051591 1-137438953478' 2024-04-25T07:53:34.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T07:53:34.209 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T07:53:34.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=150323855365 2024-04-25T07:53:34.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 150323855365 2024-04-25T07:53:34.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-124554051591 1-137438953478 2-150323855365' 2024-04-25T07:53:34.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:53:34.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-124554051591 2024-04-25T07:53:34.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:53:34.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T07:53:34.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-124554051591 2024-04-25T07:53:34.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:53:34.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051591 2024-04-25T07:53:34.358 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 124554051591 2024-04-25T07:53:34.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 124554051591' 2024-04-25T07:53:34.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T07:53:34.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051591 -lt 124554051591 2024-04-25T07:53:34.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:53:34.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:53:34.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-137438953478 2024-04-25T07:53:34.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T07:53:34.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-137438953478 2024-04-25T07:53:34.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:53:34.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953478 2024-04-25T07:53:34.805 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 137438953478 2024-04-25T07:53:34.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 137438953478' 2024-04-25T07:53:34.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:53:35.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953476 -lt 137438953478 2024-04-25T07:53:35.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T07:53:36.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T07:53:36.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T07:53:36.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953478 -lt 137438953478 2024-04-25T07:53:36.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T07:53:36.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-150323855365 2024-04-25T07:53:36.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T07:53:36.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T07:53:36.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-150323855365 2024-04-25T07:53:36.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T07:53:36.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=150323855365 2024-04-25T07:53:36.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 150323855365' 2024-04-25T07:53:36.657 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 150323855365 2024-04-25T07:53:36.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T07:53:37.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 150323855365 -lt 150323855365 2024-04-25T07:53:37.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T07:53:37.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:53:37.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:53:37.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 50 == 0 2024-04-25T07:53:37.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:53:37.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:53:37.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:53:37.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:53:37.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:53:37.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:53:37.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:53:38.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=49 2024-04-25T07:53:38.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:53:38.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:53:38.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:53:38.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 49 = 50 2024-04-25T07:53:38.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 49 '!=' -1 2024-04-25T07:53:38.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T07:53:38.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=49 2024-04-25T07:53:38.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T07:53:38.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T07:53:38.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T07:53:38.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T07:53:38.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T07:53:38.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T07:53:38.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T07:53:38.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T07:53:38.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T07:53:38.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T07:53:39.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=50 2024-04-25T07:53:39.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T07:53:39.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T07:53:39.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 50 = 50 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-force-create-pg.sh:17: run: teardown td/osd-force-create-pg 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:53:39.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:53:39.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:53:39.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:53:39.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:53:39.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:53:39.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:53:39.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:53:39.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:53:39.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:53:39.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:53:39.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:53:39.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:53:39.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:53:39.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:53:39.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:53:39.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:53:39.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:39.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:53:39.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168814 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-force-create-pg 0 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-force-create-pg 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-force-create-pg KILL 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:53:39.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:53:39.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:53:39.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:53:39.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:53:39.974 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:53:39.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:53:39.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:53:39.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:53:39.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:53:39.977 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:53:39.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:53:39.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:53:39.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:53:39.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:53:39.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:53:39.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T07:53:39.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-force-create-pg 2024-04-25T07:53:39.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:53:39.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:39.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.168814 2024-04-25T07:53:39.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.168814 2024-04-25T07:53:39.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:53:39.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:53:39.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T07:53:39.984 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T07:53:39.985 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T07:53:40.036 INFO:tasks.workunit:Running workunit osd/osd-markdown.sh... 2024-04-25T07:53:40.037 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-markdown.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh 2024-04-25T07:53:40.090 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-markdown 2024-04-25T07:53:40.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:22: run: local dir=td/osd-markdown 2024-04-25T07:53:40.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:23: run: shift 2024-04-25T07:53:40.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: export CEPH_MON=127.0.0.1:7108 2024-04-25T07:53:40.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:25: run: CEPH_MON=127.0.0.1:7108 2024-04-25T07:53:40.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:26: run: export CEPH_ARGS 2024-04-25T07:53:40.096 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: uuidgen 2024-04-25T07:53:40.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:27: run: CEPH_ARGS+='--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none ' 2024-04-25T07:53:40.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7108 ' 2024-04-25T07:53:40.096 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: set 2024-04-25T07:53:40.096 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T07:53:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:30: run: local 'funcs=TEST_markdown_boot 2024-04-25T07:53:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_markdown_boot_exceed_time 2024-04-25T07:53:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_markdown_exceed_maxdown_count 2024-04-25T07:53:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_osd_stop' 2024-04-25T07:53:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:53:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:53:40.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:53:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:53:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:53:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:53:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:53:40.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:53:40.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:53:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:53:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:53:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:53:40.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:53:40.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:53:40.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:53:40.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:53:40.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:53:40.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:53:40.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:53:40.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:53:40.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:53:40.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:53:40.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:53:40.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:53:40.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:53:40.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:53:40.109 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:53:40.109 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:40.109 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:53:40.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178330 2024-04-25T07:53:40.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:53:40.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:53:40.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:53:40.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:53:40.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:40.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:53:40.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.178330 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot td/osd-markdown 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:82: TEST_markdown_boot: local dir=td/osd-markdown 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:84: TEST_markdown_boot: run_mon td/osd-markdown a 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:53:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:53:40.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:53:40.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:53:40.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:53:40.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:53:40.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:53:40.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:53:40.628 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:53:40.628 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:40.628 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:53:40.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:53:40.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:53:40.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:53:40.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:53:40.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:53:40.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:53:40.872 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178330/ceph-mon.a.asok 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:53:40.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178330/ceph-mon.a.asok config get fsid 2024-04-25T07:53:40.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:53:40.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:53:40.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:53:40.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:53:40.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:53:40.989 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:53:40.989 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:53:40.990 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:53:40.990 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:53:40.990 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:40.990 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:53:40.990 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178330/ceph-mon.a.asok 2024-04-25T07:53:40.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:53:40.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178330/ceph-mon.a.asok config get mon_host 2024-04-25T07:53:41.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:85: TEST_markdown_boot: run_mgr td/osd-markdown x 2024-04-25T07:53:41.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:53:41.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:53:41.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:53:41.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:53:41.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:53:41.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:53:42.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:53:42.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:53:42.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:53:42.036 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:53:42.036 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:42.036 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:53:42.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:53:42.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:53:42.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:86: TEST_markdown_boot: run_osd td/osd-markdown 0 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:53:42.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:53:42.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:53:42.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:53:42.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:53:42.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:53:42.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:53:42.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:53:42.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:53:42.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:53:42.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:53:42.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=146166c3-1191-4f50-b30f-c6702059b179 2024-04-25T07:53:42.073 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 146166c3-1191-4f50-b30f-c6702059b179 2024-04-25T07:53:42.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 146166c3-1191-4f50-b30f-c6702059b179' 2024-04-25T07:53:42.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:53:42.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGDCpmUBhqBRAAj57J4VQ/jO5UM7/yPH4vvg== 2024-04-25T07:53:42.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGDCpmUBhqBRAAj57J4VQ/jO5UM7/yPH4vvg=="}' 2024-04-25T07:53:42.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 146166c3-1191-4f50-b30f-c6702059b179 -i td/osd-markdown/0/new.json 2024-04-25T07:53:42.459 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:53:42.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:53:42.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGDCpmUBhqBRAAj57J4VQ/jO5UM7/yPH4vvg== --osd-uuid 146166c3-1191-4f50-b30f-c6702059b179 2024-04-25T07:53:42.510 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:42.514+0000 7f6739cc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:42.545 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:42.546+0000 7f6739cc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:42.546 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:42.550+0000 7f6739cc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:42.547 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:42.550+0000 7f6739cc17c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:53:42.547 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:42.550+0000 7f6739cc17c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:53:42.547 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:42.550+0000 7f6739cc17c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:53:42.628 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:42.630+0000 7f6739cc17c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:53:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:53:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:53:50.523 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:53:50.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:53:50.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:53:51.075 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:53:51.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:53:51.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:53:51.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:53:51.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:53:51.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:53:51.105 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:51.105+0000 7f2171da37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:51.107 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:51.109+0000 7f2171da37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:51.108 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:51.109+0000 7f2171da37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:53:51.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:53:51.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:53:51.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:53:51.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:53:51.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:53:51.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:51.484 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:53:51.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:53:51.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:51.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:51.653 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:51.653+0000 7f2171da37c0 -1 Falling back to public interface 2024-04-25T07:53:51.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:52.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:52.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:52.875 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:53:52.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:53:52.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:52.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:53.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:54.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:54.284 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:53:54.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:54.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:53:54.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:54.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:54.497 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:53:54.497+0000 7f2171da37c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:53:54.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:55.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:55.767 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:53:55.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:55.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:53:55.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:55.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:56.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:57.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:57.375 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:53:57.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:57.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:53:57.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:57.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:57.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:53:58.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:53:58.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:53:58.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:53:58.970 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:53:58.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:53:58.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:53:59.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:00.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:00.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:00.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:54:00.420 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:54:00.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:00.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:00.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:01.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:01.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:01.955 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:54:01.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:54:01.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:01.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:02.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:03.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:03.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:03.400 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:54:03.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:54:03.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:03.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:03.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:04.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:04.939 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:54:04.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:04.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:54:04.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:04.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:05.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:06.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:06.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:06.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:54:06.453 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:54:06.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:06.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:07.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:08.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:08.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:08.093 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:54:08.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:54:08.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:08.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:08.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:09.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:09.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:09.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:54:09.514 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:54:09.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:09.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:10.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:11.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:11.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:11.182 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:54:11.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:54:11.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:11.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:11.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:12.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:12.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:12.723 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:54:12.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:54:12.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:12.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:13.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:14.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:14.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:14.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:54:14.216 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:54:14.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:14.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:14.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:15.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:15.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:15.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:54:15.820 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:54:15.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:15.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:16.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:17.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:17.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:17.875 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:54:17.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:54:17.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:17.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:18.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:19.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:19.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:19.840 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:54:19.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:54:19.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:19.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:20.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:21.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:21.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:21.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:54:21.640 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:54:21.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:21.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:22.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:23.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:23.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:23.455 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:54:23.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:54:23.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:23.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:24.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:25.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:25.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:25.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:54:25.502 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:54:25.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:25.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:26.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:27.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:27.169 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:54:27.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:27.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:54:27.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:27.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:54:27.604 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56576280,v1:127.0.0.1:6803/56576280] [v2:127.0.0.1:6804/56576280,v1:127.0.0.1:6805/56576280] exists,up 146166c3-1191-4f50-b30f-c6702059b179 2024-04-25T07:54:27.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:54:27.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:54:27.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:87: TEST_markdown_boot: run_osd td/osd-markdown 1 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:54:27.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:54:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:54:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:54:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:54:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:54:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:54:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:54:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:54:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:54:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:54:27.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:54:27.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eae3591a-60d4-4812-9112-9e19623e78c4 2024-04-25T07:54:27.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 eae3591a-60d4-4812-9112-9e19623e78c4' 2024-04-25T07:54:27.612 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 eae3591a-60d4-4812-9112-9e19623e78c4 2024-04-25T07:54:27.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:54:27.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAzDCpm6q7CJRAAsUF9+qjfAxRfv6HKzSpb4w== 2024-04-25T07:54:27.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAzDCpm6q7CJRAAsUF9+qjfAxRfv6HKzSpb4w=="}' 2024-04-25T07:54:27.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eae3591a-60d4-4812-9112-9e19623e78c4 -i td/osd-markdown/1/new.json 2024-04-25T07:54:28.206 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:54:28.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:54:28.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzDCpm6q7CJRAAsUF9+qjfAxRfv6HKzSpb4w== --osd-uuid eae3591a-60d4-4812-9112-9e19623e78c4 2024-04-25T07:54:28.259 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:28.256+0000 7f69411567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:28.259 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:28.256+0000 7f69411567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:28.259 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:28.260+0000 7f69411567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:28.259 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:28.260+0000 7f69411567c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:54:28.259 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:28.260+0000 7f69411567c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:54:28.259 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:28.260+0000 7f69411567c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:54:28.323 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:28.324+0000 7f69411567c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:54:36.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:54:36.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:54:36.560 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:54:36.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:54:36.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:54:37.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:54:37.148 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:54:37.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:54:37.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:54:37.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:54:37.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:54:37.178 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:37.176+0000 7fd4d56957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:37.180 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:37.180+0000 7fd4d56957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:37.181 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:37.180+0000 7fd4d56957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:54:37.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:54:37.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:54:37.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:54:37.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:54:37.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:54:37.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:37.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:54:37.550 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:54:37.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:37.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:37.724 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:37.724+0000 7fd4d56957c0 -1 Falling back to public interface 2024-04-25T07:54:37.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:38.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:38.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:38.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:54:38.947 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:54:38.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:38.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:39.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:40.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:40.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:40.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:54:40.374 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:54:40.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:40.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:40.547 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:54:40.548+0000 7fd4d56957c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:54:40.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:41.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:41.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:41.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:54:41.874 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:54:41.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:41.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:42.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:43.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:43.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:43.384 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:54:43.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:54:43.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:43.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:44.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:45.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:45.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:45.071 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:54:45.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:54:45.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:45.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:46.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:46.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:46.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:54:46.653 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:54:46.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:46.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:47.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:48.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:48.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:48.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:54:48.281 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:54:48.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:48.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:48.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:49.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:49.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:49.908 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:54:49.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:54:49.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:49.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:50.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:51.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:51.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:51.364 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:54:51.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:54:51.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:51.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:52.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:53.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:53.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:53.059 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:54:53.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:54:53.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:53.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:53.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:54.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:54.705 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:54:54.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:54.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:54:54.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:54.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:55.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:56.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:56.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:56.179 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:54:56.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:54:56.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:56.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:56.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:57.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:57.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:57.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:54:57.690 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:54:57.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:57.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:58.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:54:59.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:54:59.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:54:59.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:54:59.177 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:54:59.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:54:59.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:54:59.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:00.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:00.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:00.728 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:55:00.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:55:00.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:00.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:01.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:02.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:02.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:02.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:55:02.499 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:55:02.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:02.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:03.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:04.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:04.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:04.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:55:04.535 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:55:04.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:04.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:05.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:06.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:06.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:06.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:55:06.282 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:55:06.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:06.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:07.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:08.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:08.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:08.048 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:55:08.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:55:08.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:08.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:09.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:10.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:10.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:10.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:55:10.132 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:55:10.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:10.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:12.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:12.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:12.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:55:12.069 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:55:12.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:12.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:12.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:13.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:13.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:13.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:55:13.808 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:55:13.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:13.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:55:14.313 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2529837414,v1:127.0.0.1:6811/2529837414] [v2:127.0.0.1:6812/2529837414,v1:127.0.0.1:6813/2529837414] exists,up eae3591a-60d4-4812-9112-9e19623e78c4 2024-04-25T07:55:14.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:55:14.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:55:14.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:55:14.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:88: TEST_markdown_boot: run_osd td/osd-markdown 2 2024-04-25T07:55:14.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:55:14.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:55:14.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:55:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:55:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:55:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:55:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:55:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:55:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:55:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:55:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:55:14.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:55:14.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:55:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:55:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:55:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:55:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:55:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:55:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:55:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:55:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:55:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:55:14.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:55:14.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc844854-7795-4acb-ae7c-58ae497c5387 2024-04-25T07:55:14.320 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 dc844854-7795-4acb-ae7c-58ae497c5387 2024-04-25T07:55:14.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 dc844854-7795-4acb-ae7c-58ae497c5387' 2024-04-25T07:55:14.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:55:14.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiDCpmH7ItFBAAjgfXF5z3ijv0nfGTH54B0A== 2024-04-25T07:55:14.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiDCpmH7ItFBAAjgfXF5z3ijv0nfGTH54B0A=="}' 2024-04-25T07:55:14.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc844854-7795-4acb-ae7c-58ae497c5387 -i td/osd-markdown/2/new.json 2024-04-25T07:55:14.954 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:55:14.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:55:14.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiDCpmH7ItFBAAjgfXF5z3ijv0nfGTH54B0A== --osd-uuid dc844854-7795-4acb-ae7c-58ae497c5387 2024-04-25T07:55:15.003 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:15.003+0000 7f1763edd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:15.005 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:15.003+0000 7f1763edd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:15.006 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:15.007+0000 7f1763edd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:15.007 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:15.007+0000 7f1763edd7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:55:15.007 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:15.007+0000 7f1763edd7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:55:15.007 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:15.007+0000 7f1763edd7c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:55:15.071 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:15.071+0000 7f1763edd7c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:55:23.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:55:23.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:55:23.580 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:55:23.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:55:23.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:55:24.148 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:55:24.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:55:24.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:55:24.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:55:24.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:55:24.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:55:24.172 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:24.171+0000 7f08dd6057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:24.174 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:24.175+0000 7f08dd6057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:24.175 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:24.175+0000 7f08dd6057c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:55:24.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:55:24.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:55:24.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:55:24.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:55:24.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:55:24.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:24.560 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:55:24.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:55:24.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:24.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:24.724 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:24.723+0000 7f08dd6057c0 -1 Falling back to public interface 2024-04-25T07:55:25.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:26.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:26.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:26.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:55:26.016 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:55:26.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:26.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:26.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:27.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:27.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:27.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:55:27.422 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:55:27.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:27.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:27.722 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:55:27.723+0000 7f08dd6057c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:55:27.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:28.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:28.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:28.852 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:55:28.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:55:28.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:28.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:29.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:30.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:30.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:30.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:55:30.278 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:55:30.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:30.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:30.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:31.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:31.771 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:55:31.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:31.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:55:31.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:31.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:32.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:33.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:33.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:33.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:55:33.330 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:55:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:33.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:55:34.839 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:55:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:34.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:35.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:36.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:36.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:36.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:55:36.602 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:55:36.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:36.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:37.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:38.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:38.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:38.229 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:55:38.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:55:38.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:38.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:38.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:39.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:39.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:39.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:55:39.716 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:55:39.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:39.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:40.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:41.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:41.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:41.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:55:41.389 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:55:41.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:41.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:42.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:43.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:43.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:43.079 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:55:43.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:55:43.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:43.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:43.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:44.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:44.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:44.675 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:55:44.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:55:44.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:44.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:45.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:46.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:46.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:46.348 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:55:46.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:55:46.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:46.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:46.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:47.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:47.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:47.847 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:55:47.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:55:47.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:47.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:48.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:49.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:49.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:49.754 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:55:49.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:55:49.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:49.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:50.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:51.845 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:55:51.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:55:51.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:51.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:52.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:55:53.925 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:55:53.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:53.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:54.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:55.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:55.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:55.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:55:55.995 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:55:55.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:55.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:57.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:58.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:58.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:58.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:55:58.007 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:55:58.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:58.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:55:58.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:55:59.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:55:59.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:55:59.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:55:59.918 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:55:59.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:55:59.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:00.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:56:01.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:56:01.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:56:01.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:56:01.448 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:56:01.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:56:01.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:56:01.873 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3086875984,v1:127.0.0.1:6819/3086875984] [v2:127.0.0.1:6820/3086875984,v1:127.0.0.1:6821/3086875984] exists,up dc844854-7795-4acb-ae7c-58ae497c5387 2024-04-25T07:56:01.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:56:01.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:56:01.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:56:01.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:90: TEST_markdown_boot: create_rbd_pool 2024-04-25T07:56:01.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:56:02.454 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' does not exist 2024-04-25T07:56:02.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:56:02.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:56:03.254 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' already exists 2024-04-25T07:56:03.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:56:04.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:56:07.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:93: TEST_markdown_boot: local count=3 2024-04-25T07:56:07.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:94: TEST_markdown_boot: local sleeptime=10 2024-04-25T07:56:07.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:95: TEST_markdown_boot: local period=120 2024-04-25T07:56:07.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:96: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:56:07.184 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T07:56:07.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:97: TEST_markdown_boot: ceph tell osd.0 injectargs '--osd_max_markdown_period 120' 2024-04-25T07:56:07.385 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T07:56:07.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:99: TEST_markdown_boot: markdown_N_impl 3 120 10 2024-04-25T07:56:07.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=3 2024-04-25T07:56:07.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=120 2024-04-25T07:56:07.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:56:07.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 3 2024-04-25T07:56:07.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:56:07.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:56:07.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:56:07.954 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:56:07.954 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T07:56:07.954 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T07:56:07.954 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:07.954 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:56:07.954 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:56:07.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:56:07.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:56:07.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:56:08.361 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:08.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:56:08.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:56:08.867 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T07:56:08.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:56:18.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:56:18.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:56:19.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:56:19.552 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:56:19.552 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T07:56:19.552 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T07:56:19.552 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:19.552 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:56:19.552 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:56:19.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:56:19.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:56:19.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:56:20.025 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:20.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:56:20.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:56:20.547 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T07:56:20.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:56:30.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:56:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:56:30.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:56:31.121 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:56:31.122 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T07:56:31.122 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T07:56:31.122 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:31.122 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:56:31.122 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:56:31.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:56:31.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:56:31.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:56:31.544 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:31.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:56:31.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:56:32.006 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T07:56:32.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:56:42.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:101: TEST_markdown_boot: sleep 15 2024-04-25T07:56:57.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:102: TEST_markdown_boot: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:56:57.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: ceph osd tree 2024-04-25T07:56:57.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep up 2024-04-25T07:56:57.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:103: TEST_markdown_boot: grep osd.0 2024-04-25T07:56:57.632 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:56:57.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T07:56:57.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:56:57.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:56:57.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:56:57.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:56:57.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:56:57.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:56:57.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:56:57.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:56:57.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:56:57.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:56:57.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:56:57.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:56:57.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:56:57.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:56:57.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:56:57.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:56:57.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:56:57.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:56:57.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:56:57.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:56:57.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:56:57.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:56:57.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:56:57.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:56:57.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:56:57.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178330 2024-04-25T07:56:57.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:56:57.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:56:57.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T07:56:57.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T07:56:57.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T07:56:57.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T07:56:57.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T07:56:57.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T07:56:57.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T07:56:57.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T07:56:57.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T07:56:57.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T07:56:57.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:56:57.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T07:56:57.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T07:56:57.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T07:56:57.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T07:56:57.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T07:56:57.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T07:56:57.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T07:56:57.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:56:57.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:56:57.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178330 2024-04-25T07:56:57.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T07:56:57.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T07:56:57.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T07:56:57.994 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T07:56:57.994 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:56:57.994 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:56:57.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.178330 2024-04-25T07:56:57.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_boot_exceed_time td/osd-markdown 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:107: TEST_markdown_boot_exceed_time: local dir=td/osd-markdown 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:109: TEST_markdown_boot_exceed_time: run_mon td/osd-markdown a 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T07:56:57.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T07:56:58.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T07:56:58.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:56:58.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:56:58.556 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:56:58.556 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:56:58.556 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:56:58.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:56:58.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T07:56:58.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T07:56:58.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T07:56:58.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:56:58.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:56:58.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T07:56:58.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T07:56:58.810 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:56:58.810 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:56:58.811 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:56:58.811 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:56:58.811 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:56:58.811 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:56:58.811 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178330/ceph-mon.a.asok 2024-04-25T07:56:58.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:56:58.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178330/ceph-mon.a.asok config get fsid 2024-04-25T07:56:58.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T07:56:58.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T07:56:58.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T07:56:58.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T07:56:58.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T07:56:58.922 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T07:56:58.922 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T07:56:58.923 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T07:56:58.923 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T07:56:58.923 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:56:58.924 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:56:58.924 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178330/ceph-mon.a.asok 2024-04-25T07:56:58.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T07:56:58.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178330/ceph-mon.a.asok config get mon_host 2024-04-25T07:56:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:110: TEST_markdown_boot_exceed_time: run_mgr td/osd-markdown x 2024-04-25T07:56:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T07:56:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T07:56:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T07:56:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T07:56:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T07:56:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T07:57:00.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T07:57:00.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:57:00.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:57:00.763 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:57:00.763 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:00.763 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:57:00.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:57:00.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:57:00.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T07:57:00.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:111: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 0 2024-04-25T07:57:00.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:57:00.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:57:00.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:57:00.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:57:00.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:57:00.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:57:00.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:57:00.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:57:00.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:57:00.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:57:00.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T07:57:00.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:57:00.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e493428a-a670-4827-8946-de09cd814a7a 2024-04-25T07:57:00.801 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 e493428a-a670-4827-8946-de09cd814a7a 2024-04-25T07:57:00.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e493428a-a670-4827-8946-de09cd814a7a' 2024-04-25T07:57:00.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:57:00.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDMDCpmyPO4MBAAm7Gn1UImouFLZDr01Jgi0A== 2024-04-25T07:57:00.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDMDCpmyPO4MBAAm7Gn1UImouFLZDr01Jgi0A=="}' 2024-04-25T07:57:00.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e493428a-a670-4827-8946-de09cd814a7a -i td/osd-markdown/0/new.json 2024-04-25T07:57:01.170 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:57:01.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T07:57:01.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMDCpmyPO4MBAAm7Gn1UImouFLZDr01Jgi0A== --osd-uuid e493428a-a670-4827-8946-de09cd814a7a 2024-04-25T07:57:01.216 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:01.216+0000 7ff4ae4c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:01.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:01.256+0000 7ff4ae4c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:01.257 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:01.256+0000 7ff4ae4c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:01.257 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:01.256+0000 7ff4ae4c37c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:57:01.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:01.256+0000 7ff4ae4c37c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:57:01.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:01.256+0000 7ff4ae4c37c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T07:57:01.295 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:01.296+0000 7ff4ae4c37c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T07:57:09.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T07:57:09.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:57:09.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T07:57:09.033 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T07:57:09.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:57:09.587 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T07:57:09.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T07:57:09.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:57:09.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:57:09.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:57:09.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:57:09.618 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:09.616+0000 7fead9b2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:09.620 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:09.620+0000 7fead9b2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:09.621 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:09.620+0000 7fead9b2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:10.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T07:57:10.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:57:10.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T07:57:10.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:57:10.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:57:10.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:10.001 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:57:10.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:57:10.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:10.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:10.189 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:10.188+0000 7fead9b2f7c0 -1 Falling back to public interface 2024-04-25T07:57:10.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:11.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:11.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:11.393 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:57:11.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:57:11.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:11.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:11.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:12.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:12.799 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:57:12.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:12.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:57:12.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:12.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:13.008 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:13.008+0000 7fead9b2f7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T07:57:13.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:14.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:14.261 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:57:14.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:14.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:57:14.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:14.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:14.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:15.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:15.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:15.707 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:57:15.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:57:15.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:15.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:16.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:17.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:17.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:17.235 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:57:17.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:57:17.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:17.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:17.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:18.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:18.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:18.750 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:57:18.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:57:18.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:18.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:19.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:20.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:20.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:20.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:57:20.252 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:57:20.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:20.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:20.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:21.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:21.708 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:57:21.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:21.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:57:21.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:21.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:22.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:23.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:23.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:23.163 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:57:23.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:57:23.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:23.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:23.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:24.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:24.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:24.614 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:57:24.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:57:24.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:24.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:25.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:26.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:26.432 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:57:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:57:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:26.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:27.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:28.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:28.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:28.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:57:28.091 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:57:28.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:28.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:28.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:29.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:29.602 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:57:29.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:29.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:57:29.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:29.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:30.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:31.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:31.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:31.160 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:57:31.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:57:31.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:31.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:31.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:32.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:32.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:32.689 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:57:32.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:57:32.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:32.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:33.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:34.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:34.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:34.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:57:34.883 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:57:34.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:34.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:35.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:36.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:36.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:36.823 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:57:36.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:57:36.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:36.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:37.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:38.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:38.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:38.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:57:38.664 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:57:38.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:38.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:39.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:40.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:40.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:40.460 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:57:40.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:57:40.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:40.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:41.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:42.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:42.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:42.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:57:42.351 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:57:42.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:42.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:43.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:44.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:44.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:44.082 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:57:44.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:57:44.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:44.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:45.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:46.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:46.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:46.097 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:57:46.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:57:46.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:46.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T07:57:46.518 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1401662003,v1:127.0.0.1:6803/1401662003] [v2:127.0.0.1:6804/1401662003,v1:127.0.0.1:6805/1401662003] exists,up e493428a-a670-4827-8946-de09cd814a7a 2024-04-25T07:57:46.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:57:46.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:57:46.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:57:46.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:112: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 1 2024-04-25T07:57:46.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:57:46.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:57:46.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T07:57:46.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:57:46.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:57:46.522 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:57:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T07:57:46.524 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:57:46.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9c091b9b-5e28-4f24-9358-af0737297e2c 2024-04-25T07:57:46.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9c091b9b-5e28-4f24-9358-af0737297e2c' 2024-04-25T07:57:46.524 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 9c091b9b-5e28-4f24-9358-af0737297e2c 2024-04-25T07:57:46.524 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:57:46.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD6DCpmetQrIBAAS9ToHz+76z8mmXABgF9rjg== 2024-04-25T07:57:46.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD6DCpmetQrIBAAS9ToHz+76z8mmXABgF9rjg=="}' 2024-04-25T07:57:46.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9c091b9b-5e28-4f24-9358-af0737297e2c -i td/osd-markdown/1/new.json 2024-04-25T07:57:47.113 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:57:47.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T07:57:47.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD6DCpmetQrIBAAS9ToHz+76z8mmXABgF9rjg== --osd-uuid 9c091b9b-5e28-4f24-9358-af0737297e2c 2024-04-25T07:57:47.157 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:47.155+0000 7fcd45ba47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:47.160 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:47.159+0000 7fcd45ba47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:47.161 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:47.159+0000 7fcd45ba47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:47.161 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:47.159+0000 7fcd45ba47c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:57:47.161 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:47.159+0000 7fcd45ba47c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:57:47.161 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:47.159+0000 7fcd45ba47c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T07:57:47.238 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:47.235+0000 7fcd45ba47c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T07:57:55.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T07:57:55.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:57:55.600 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T07:57:55.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T07:57:55.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:57:56.197 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T07:57:56.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T07:57:56.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:57:56.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:57:56.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:57:56.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:57:56.227 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:56.223+0000 7f2fb4fc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:56.229 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:56.227+0000 7f2fb4fc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:56.230 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:56.227+0000 7f2fb4fc87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:57:56.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T07:57:56.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:57:56.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T07:57:56.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:57:56.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:57:56.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:56.592 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:57:56.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:57:56.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:56.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:57:56.769 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:56.767+0000 7f2fb4fc87c0 -1 Falling back to public interface 2024-04-25T07:57:56.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:58.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:58.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:58.001 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:57:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:57:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:57:58.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:57:59.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:57:59.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:57:59.408 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:57:59.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:57:59.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:57:59.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:57:59.661 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:57:59.659+0000 7f2fb4fc87c0 -1 osd.1 0 log_to_monitors true 2024-04-25T07:57:59.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:00.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:00.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:00.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:58:00.834 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:58:00.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:00.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:01.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:02.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:02.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:02.366 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:58:02.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:58:02.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:02.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:02.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:03.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:03.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:03.969 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:58:03.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:58:03.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:03.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:04.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:05.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:05.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:05.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:58:05.460 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:58:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:06.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:07.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:07.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:07.108 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:58:07.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:58:07.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:07.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:07.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:08.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:08.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:08.589 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:58:08.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:58:08.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:08.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:09.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:10.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:10.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:10.207 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:58:10.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:58:10.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:10.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:10.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:11.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:11.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:11.863 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:58:11.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:58:11.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:11.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:12.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:13.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:13.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:13.377 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:58:13.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:58:13.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:13.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:13.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:14.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:14.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:14.913 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:58:14.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:58:14.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:14.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:15.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:16.358 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:58:16.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:58:16.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:16.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:16.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:17.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:17.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:17.883 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:58:17.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:58:17.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:17.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:18.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:19.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:19.519 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:58:19.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:19.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:58:19.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:19.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:20.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:21.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:21.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:21.457 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:58:21.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:58:21.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:21.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:22.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:23.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:23.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:23.415 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:58:23.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:58:23.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:23.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:24.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:25.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:25.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:25.350 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:58:25.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:58:25.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:25.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:26.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:27.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:27.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:27.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:58:27.230 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:58:27.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:27.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:28.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:29.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:29.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:29.311 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:58:29.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:58:29.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:29.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:30.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:31.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:31.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:31.205 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:58:31.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:58:31.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:31.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:31.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:32.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:32.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:32.925 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:58:32.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:58:32.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:32.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T07:58:33.392 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3660141610,v1:127.0.0.1:6811/3660141610] [v2:127.0.0.1:6812/3660141610,v1:127.0.0.1:6813/3660141610] exists,up 9c091b9b-5e28-4f24-9358-af0737297e2c 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:113: TEST_markdown_boot_exceed_time: run_osd td/osd-markdown 2 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T07:58:33.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T07:58:33.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T07:58:33.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T07:58:33.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T07:58:33.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=772f32a0-0584-4a75-b4e8-820135d69018 2024-04-25T07:58:33.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 772f32a0-0584-4a75-b4e8-820135d69018' 2024-04-25T07:58:33.398 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 772f32a0-0584-4a75-b4e8-820135d69018 2024-04-25T07:58:33.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T07:58:33.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApDSpmiS3uGBAAU0M9JdoDP/r4zEVbfsQ5xw== 2024-04-25T07:58:33.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApDSpmiS3uGBAAU0M9JdoDP/r4zEVbfsQ5xw=="}' 2024-04-25T07:58:33.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 772f32a0-0584-4a75-b4e8-820135d69018 -i td/osd-markdown/2/new.json 2024-04-25T07:58:34.028 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:58:34.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T07:58:34.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApDSpmiS3uGBAAU0M9JdoDP/r4zEVbfsQ5xw== --osd-uuid 772f32a0-0584-4a75-b4e8-820135d69018 2024-04-25T07:58:34.080 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:34.078+0000 7f59d9b397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:34.082 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:34.078+0000 7f59d9b397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:34.084 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:34.082+0000 7f59d9b397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:34.084 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:34.082+0000 7f59d9b397c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:58:34.084 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:34.082+0000 7f59d9b397c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:58:34.085 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:34.082+0000 7f59d9b397c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T07:58:34.170 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:34.166+0000 7f59d9b397c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T07:58:42.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T07:58:42.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T07:58:42.376 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T07:58:42.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T07:58:42.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T07:58:42.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T07:58:42.955 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T07:58:42.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T07:58:42.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T07:58:42.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T07:58:42.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T07:58:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:42.974+0000 7ff6784d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:42.981 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:42.978+0000 7ff6784d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:42.982 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:42.978+0000 7ff6784d67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T07:58:43.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T07:58:43.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T07:58:43.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T07:58:43.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T07:58:43.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T07:58:43.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:43.352 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T07:58:43.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T07:58:43.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:43.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:43.524 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:43.522+0000 7ff6784d67c0 -1 Falling back to public interface 2024-04-25T07:58:43.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:44.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:44.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:44.754 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T07:58:44.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T07:58:44.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:44.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:45.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:46.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:46.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:46.163 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T07:58:46.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T07:58:46.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:46.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:46.439 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T07:58:46.434+0000 7ff6784d67c0 -1 osd.2 0 log_to_monitors true 2024-04-25T07:58:46.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:47.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:47.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:47.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T07:58:47.577 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T07:58:47.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:47.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:48.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:49.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:49.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:49.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T07:58:49.157 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T07:58:49.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:49.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:49.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:50.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:50.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:50.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T07:58:50.876 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T07:58:50.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:50.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:51.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:52.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:52.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:52.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T07:58:52.299 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T07:58:52.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:52.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:52.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:53.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:53.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:53.981 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T07:58:53.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T07:58:53.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:53.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:54.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:55.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:55.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:55.584 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T07:58:55.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T07:58:55.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:55.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:56.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:57.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:57.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:57.376 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T07:58:57.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T07:58:57.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:57.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:58:58.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:58:59.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:58:59.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:58:59.276 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T07:58:59.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T07:58:59.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:58:59.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:00.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:01.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:01.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:01.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T07:59:01.004 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T07:59:01.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:01.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:01.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:02.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:02.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:02.460 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T07:59:02.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T07:59:02.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:02.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:03.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:04.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:04.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:04.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T07:59:04.093 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T07:59:04.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:04.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:04.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:05.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:05.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:05.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T07:59:05.579 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T07:59:05.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:05.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:06.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:07.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:07.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:07.004 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T07:59:07.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T07:59:07.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:07.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:07.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:08.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:08.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:08.599 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T07:59:08.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T07:59:08.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:08.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:09.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:10.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:10.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:10.372 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T07:59:10.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T07:59:10.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:10.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:11.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:12.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:12.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:12.398 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T07:59:12.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T07:59:12.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:12.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:13.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:14.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:14.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:14.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T07:59:14.450 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T07:59:14.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:14.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:15.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:16.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:16.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:16.304 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T07:59:16.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T07:59:16.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:16.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:17.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:18.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:18.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:18.161 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T07:59:18.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T07:59:18.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:18.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:19.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T07:59:20.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T07:59:20.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T07:59:20.154 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T07:59:20.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T07:59:20.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T07:59:20.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T07:59:20.655 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2564941801,v1:127.0.0.1:6819/2564941801] [v2:127.0.0.1:6820/2564941801,v1:127.0.0.1:6821/2564941801] exists,up 772f32a0-0584-4a75-b4e8-820135d69018 2024-04-25T07:59:20.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T07:59:20.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T07:59:20.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T07:59:20.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:115: TEST_markdown_boot_exceed_time: create_rbd_pool 2024-04-25T07:59:20.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T07:59:21.090 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' does not exist 2024-04-25T07:59:21.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T07:59:21.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T07:59:21.822 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' already exists 2024-04-25T07:59:21.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T07:59:22.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:118: TEST_markdown_boot_exceed_time: local count=3 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:119: TEST_markdown_boot_exceed_time: local period=20 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:120: TEST_markdown_boot_exceed_time: local sleeptime=10 2024-04-25T07:59:25.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:121: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T07:59:25.652 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T07:59:25.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:122: TEST_markdown_boot_exceed_time: ceph tell osd.0 injectargs '--osd_max_markdown_period 20' 2024-04-25T07:59:25.809 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T07:59:25.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:124: TEST_markdown_boot_exceed_time: markdown_N_impl 4 20 10 2024-04-25T07:59:25.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-04-25T07:59:25.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=20 2024-04-25T07:59:25.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T07:59:25.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-04-25T07:59:25.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:59:25.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:59:25.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:59:26.347 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:59:26.348 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T07:59:26.348 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T07:59:26.348 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:26.348 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:59:26.348 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:59:26.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:59:26.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:59:26.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:59:26.816 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:26.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:59:26.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:59:27.246 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T07:59:27.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:59:37.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:59:37.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:59:37.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:59:37.901 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:59:37.902 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T07:59:37.902 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T07:59:37.902 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:37.902 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:59:37.902 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:59:37.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:59:37.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:59:37.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:59:38.326 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:38.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:59:38.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:59:38.877 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T07:59:38.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T07:59:48.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T07:59:48.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T07:59:49.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T07:59:49.449 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T07:59:49.449 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T07:59:49.449 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T07:59:49.449 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:49.449 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T07:59:49.449 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T07:59:49.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T07:59:49.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T07:59:49.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T07:59:49.894 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T07:59:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T07:59:49.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T07:59:50.414 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T07:59:50.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T08:00:00.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T08:00:00.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T08:00:00.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T08:00:00.985 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T08:00:00.985 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T08:00:00.985 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T08:00:00.985 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:00:00.985 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T08:00:00.985 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T08:00:01.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T08:00:01.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T08:00:01.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T08:00:01.415 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:00:01.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T08:00:01.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T08:00:01.941 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T08:00:01.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T08:00:11.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:125: TEST_markdown_boot_exceed_time: sleep 15 2024-04-25T08:00:26.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:126: TEST_markdown_boot_exceed_time: ceph tell osd.0 get_latest_osdmap 2024-04-25T08:00:27.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: ceph osd tree 2024-04-25T08:00:27.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep up 2024-04-25T08:00:27.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:127: TEST_markdown_boot_exceed_time: grep osd.0 2024-04-25T08:00:27.534 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:00:27.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T08:00:27.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:00:27.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:00:27.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:00:27.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:00:27.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:00:27.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:00:27.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:00:27.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:00:27.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:00:27.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:00:27.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:00:27.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:00:27.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:00:27.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:00:27.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:00:27.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:27.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:00:27.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:00:27.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:27.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:00:27.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:00:27.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:00:27.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:00:27.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:27.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:00:27.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178330 2024-04-25T08:00:27.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:00:27.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:00:27.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T08:00:27.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T08:00:27.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T08:00:27.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T08:00:27.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:00:27.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:00:27.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:00:27.878 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:00:27.878 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:00:27.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:00:27.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:00:27.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:00:27.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:00:27.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:00:27.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:00:27.882 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:00:27.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:00:27.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:00:27.883 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:00:27.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:27.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:00:27.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:00:27.884 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:00:27.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:00:27.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:00:27.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:00:27.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:00:27.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:27.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:00:27.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178330 2024-04-25T08:00:27.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:00:27.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:00:27.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T08:00:27.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:00:27.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:27.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:00:27.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.178330 2024-04-25T08:00:27.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_markdown_exceed_maxdown_count td/osd-markdown 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:59: TEST_markdown_exceed_maxdown_count: local dir=td/osd-markdown 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:61: TEST_markdown_exceed_maxdown_count: run_mon td/osd-markdown a 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T08:00:27.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T08:00:28.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:00:28.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:28.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:28.432 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:28.433 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:28.433 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:00:28.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:00:28.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:00:28.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:00:28.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:00:28.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:00:28.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:00:28.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:00:28.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:00:28.669 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:00:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:00:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:00:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:00:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:00:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178330/ceph-mon.a.asok 2024-04-25T08:00:28.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:00:28.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178330/ceph-mon.a.asok config get fsid 2024-04-25T08:00:28.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:00:28.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:00:28.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:00:28.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:00:28.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:00:28.791 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:00:28.791 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:00:28.791 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:00:28.791 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:00:28.792 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:28.792 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:00:28.792 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178330/ceph-mon.a.asok 2024-04-25T08:00:28.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:00:28.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178330/ceph-mon.a.asok config get mon_host 2024-04-25T08:00:28.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:62: TEST_markdown_exceed_maxdown_count: run_mgr td/osd-markdown x 2024-04-25T08:00:28.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T08:00:28.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:00:28.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:00:28.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:00:28.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T08:00:28.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:00:29.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:00:29.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:29.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:29.886 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:29.886 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:29.886 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:00:29.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:00:29.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:00:29.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:00:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:63: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 0 2024-04-25T08:00:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T08:00:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:00:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:00:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:00:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T08:00:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T08:00:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:00:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:00:29.925 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:00:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:00:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:00:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:00:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:00:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T08:00:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T08:00:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:00:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:00:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:00:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:00:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:00:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:00:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T08:00:29.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:00:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ca5d70a-4ca3-4f1c-9103-57927f399158 2024-04-25T08:00:29.926 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 6ca5d70a-4ca3-4f1c-9103-57927f399158 2024-04-25T08:00:29.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 6ca5d70a-4ca3-4f1c-9103-57927f399158' 2024-04-25T08:00:29.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:00:29.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdDSpmPRc3OBAA9ulSDjisYJyiiiWy1RDO/A== 2024-04-25T08:00:29.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdDSpmPRc3OBAA9ulSDjisYJyiiiWy1RDO/A=="}' 2024-04-25T08:00:29.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ca5d70a-4ca3-4f1c-9103-57927f399158 -i td/osd-markdown/0/new.json 2024-04-25T08:00:30.297 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:00:30.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T08:00:30.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdDSpmPRc3OBAA9ulSDjisYJyiiiWy1RDO/A== --osd-uuid 6ca5d70a-4ca3-4f1c-9103-57927f399158 2024-04-25T08:00:30.346 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:30.343+0000 7fc3a03227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:30.382 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:30.379+0000 7fc3a03227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:30.383 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:30.383+0000 7fc3a03227c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:30.385 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:30.383+0000 7fc3a03227c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T08:00:30.385 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:30.383+0000 7fc3a03227c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T08:00:30.385 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:30.383+0000 7fc3a03227c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T08:00:30.422 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:30.419+0000 7fc3a03227c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T08:00:38.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T08:00:38.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:00:38.278 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:00:38.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:00:38.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:00:38.849 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:00:38.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:00:38.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:00:38.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:00:38.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:00:38.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:00:38.879 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:38.875+0000 7fb1945487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:38.881 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:38.879+0000 7fb1945487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:38.883 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:38.879+0000 7fb1945487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:00:39.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:00:39.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:00:39.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:00:39.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:00:39.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:00:39.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:39.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:00:39.259 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:00:39.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:39.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:39.427+0000 7fb1945487c0 -1 Falling back to public interface 2024-04-25T08:00:39.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:40.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:40.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:40.655 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:00:40.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:00:40.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:40.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:41.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:42.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:42.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:42.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:00:42.057 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:00:42.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:42.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:42.241 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:00:42.239+0000 7fb1945487c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:00:42.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:43.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:43.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:43.508 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:00:43.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:00:43.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:43.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:43.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:44.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:44.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:44.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:00:44.997 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:00:44.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:44.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:45.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:46.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:46.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:46.589 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:00:46.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:00:46.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:46.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:47.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:48.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:48.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:48.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:00:48.018 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:00:48.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:48.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:48.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:49.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:49.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:49.577 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:00:49.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:00:49.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:49.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:50.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:51.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:51.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:51.011 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:00:51.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:00:51.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:51.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:51.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:52.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:52.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:52.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:00:52.681 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:00:52.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:52.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:53.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:54.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:54.168 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:00:54.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:54.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:00:54.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:54.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:54.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:55.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:55.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:55.909 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:00:55.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:00:55.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:55.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:56.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:57.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:57.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:57.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:00:57.541 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:00:57.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:57.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:58.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:00:59.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:00:59.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:00:59.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:00:59.005 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:00:59.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:00:59.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:00:59.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:00.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:00.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:00.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:01:00.538 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:01:00.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:00.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:01.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:02.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:02.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:02.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:01:02.004 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:01:02.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:02.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:03.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:04.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:04.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:04.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:01:04.680 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:01:04.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:04.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:05.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:06.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:06.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:06.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:01:06.984 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:01:06.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:06.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:09.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:09.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:09.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:01:09.172 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:01:09.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:09.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:10.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:11.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:11.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:11.528 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T08:01:11.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:01:11.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:11.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:12.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:13.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:13.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:13.610 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T08:01:13.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:01:13.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:13.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:14.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:15.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:15.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:15.355 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T08:01:15.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:01:15.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:15.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:01:15.868 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/355734262,v1:127.0.0.1:6803/355734262] [v2:127.0.0.1:6804/355734262,v1:127.0.0.1:6805/355734262] exists,up 6ca5d70a-4ca3-4f1c-9103-57927f399158 2024-04-25T08:01:15.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:01:15.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:01:15.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:01:15.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:64: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 1 2024-04-25T08:01:15.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T08:01:15.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T08:01:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:01:15.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:01:15.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:01:15.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:01:15.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:01:15.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:01:15.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T08:01:15.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:01:15.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ac7d01f4-bd1e-420e-884f-617204e4e088 2024-04-25T08:01:15.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ac7d01f4-bd1e-420e-884f-617204e4e088' 2024-04-25T08:01:15.876 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 ac7d01f4-bd1e-420e-884f-617204e4e088 2024-04-25T08:01:15.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:01:15.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLDSpmc8QGNRAAAJ8IooxD+8vo3D3G7galYw== 2024-04-25T08:01:15.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLDSpmc8QGNRAAAJ8IooxD+8vo3D3G7galYw=="}' 2024-04-25T08:01:15.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ac7d01f4-bd1e-420e-884f-617204e4e088 -i td/osd-markdown/1/new.json 2024-04-25T08:01:16.459 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:01:16.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T08:01:16.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLDSpmc8QGNRAAAJ8IooxD+8vo3D3G7galYw== --osd-uuid ac7d01f4-bd1e-420e-884f-617204e4e088 2024-04-25T08:01:16.513 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:16.510+0000 7f3ba25067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:16.516 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:16.514+0000 7f3ba25067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:16.517 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:16.514+0000 7f3ba25067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:16.518 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:16.514+0000 7f3ba25067c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T08:01:16.518 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:16.514+0000 7f3ba25067c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T08:01:16.518 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:16.514+0000 7f3ba25067c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T08:01:16.593 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:16.590+0000 7f3ba25067c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T08:01:24.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T08:01:24.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:01:24.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:01:24.607 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:01:24.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:01:25.152 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:01:25.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:01:25.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:01:25.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:01:25.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:01:25.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:01:25.182 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:25.178+0000 7fbdc55fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:25.217 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:25.214+0000 7fbdc55fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:25.218 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:25.214+0000 7fbdc55fa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:01:25.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:01:25.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:01:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:01:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:01:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:01:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:25.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:01:25.561 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:01:25.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:25.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:25.778 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:25.774+0000 7fbdc55fa7c0 -1 Falling back to public interface 2024-04-25T08:01:25.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:26.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:26.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:26.955 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:01:26.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:01:26.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:26.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:27.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:28.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:28.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:28.391 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:01:28.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:01:28.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:28.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:28.656 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:01:28.654+0000 7fbdc55fa7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:01:28.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:29.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:29.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:29.825 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:01:29.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:01:29.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:29.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:30.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:31.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:31.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:31.348 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:01:31.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:01:31.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:31.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:32.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:33.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:33.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:33.132 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:01:33.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:01:33.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:33.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:33.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:34.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:34.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:34.774 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:01:34.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:01:34.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:34.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:36.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:36.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:36.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:01:36.610 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:01:36.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:36.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:37.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:38.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:38.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:38.224 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:01:38.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:01:38.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:38.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:38.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:39.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:39.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:39.717 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:01:39.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:01:39.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:39.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:40.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:41.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:41.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:41.185 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:01:41.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:01:41.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:41.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:41.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:42.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:42.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:42.852 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:01:42.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:01:42.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:42.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:43.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:44.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:44.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:44.583 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:01:44.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:01:44.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:44.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:45.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:46.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:46.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:46.247 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:01:46.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:01:46.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:46.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:46.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:47.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:47.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:47.670 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:01:47.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:01:47.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:47.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:48.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:49.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:49.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:49.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:01:49.160 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:01:49.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:49.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:49.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:50.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:50.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:50.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:01:50.822 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:01:50.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:50.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:51.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:52.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:52.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:52.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:01:52.594 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:01:52.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:52.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:53.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:54.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:54.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:54.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:01:54.321 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:01:54.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:54.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:55.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:56.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:56.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:56.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:01:56.243 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T08:01:56.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:56.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:57.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:58.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:58.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:58.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:01:58.066 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T08:01:58.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:58.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:01:58.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:01:59.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:01:59.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:01:59.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:01:59.807 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T08:01:59.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:01:59.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:02:00.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:01.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:01.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:01.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:02:01.522 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T08:02:01.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:01.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:02:02.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:03.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:03.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:03.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 23 2024-04-25T08:02:03.012 INFO:tasks.workunit.client.0.smithi134.stdout:23 2024-04-25T08:02:03.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:03.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:02:03.491 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2985959903,v1:127.0.0.1:6811/2985959903] [v2:127.0.0.1:6812/2985959903,v1:127.0.0.1:6813/2985959903] exists,up ac7d01f4-bd1e-420e-884f-617204e4e088 2024-04-25T08:02:03.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:02:03.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:02:03.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:02:03.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:65: TEST_markdown_exceed_maxdown_count: run_osd td/osd-markdown 2 2024-04-25T08:02:03.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T08:02:03.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:02:03.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:02:03.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:02:03.496 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:02:03.496 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:02:03.496 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:02:03.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:02:03.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:02:03.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:02:03.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:02:03.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b05d606c-e545-4c38-b3d2-4bbf3083b409 2024-04-25T08:02:03.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b05d606c-e545-4c38-b3d2-4bbf3083b409' 2024-04-25T08:02:03.499 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 b05d606c-e545-4c38-b3d2-4bbf3083b409 2024-04-25T08:02:03.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:02:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7DSpmJlm8HhAAE1g/f03BvCdmkT3VL0KGQA== 2024-04-25T08:02:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7DSpmJlm8HhAAE1g/f03BvCdmkT3VL0KGQA=="}' 2024-04-25T08:02:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b05d606c-e545-4c38-b3d2-4bbf3083b409 -i td/osd-markdown/2/new.json 2024-04-25T08:02:04.102 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:02:04.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T08:02:04.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7DSpmJlm8HhAAE1g/f03BvCdmkT3VL0KGQA== --osd-uuid b05d606c-e545-4c38-b3d2-4bbf3083b409 2024-04-25T08:02:04.147 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:04.145+0000 7fc285e487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:04.149 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:04.149+0000 7fc285e487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:04.151 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:04.149+0000 7fc285e487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:04.151 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:04.153+0000 7fc285e487c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T08:02:04.151 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:04.153+0000 7fc285e487c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T08:02:04.151 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:04.153+0000 7fc285e487c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T08:02:04.228 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:04.229+0000 7fc285e487c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T08:02:12.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T08:02:12.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:02:12.730 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T08:02:12.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:02:12.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:02:13.329 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T08:02:13.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:02:13.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:02:13.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:02:13.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:02:13.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:02:13.360 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:13.361+0000 7f52bf8297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:13.362 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:13.361+0000 7f52bf8297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:13.363 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:13.365+0000 7f52bf8297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:02:13.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:02:13.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:02:13.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:02:13.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:02:13.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:02:13.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:13.736 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:02:13.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:02:13.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:13.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:13.926 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:13.925+0000 7f52bf8297c0 -1 Falling back to public interface 2024-04-25T08:02:14.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:15.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:15.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:15.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:02:15.155 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:02:15.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:15.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:15.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:16.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:16.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:16.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:02:16.563 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:02:16.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:16.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:16.789 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:02:16.789+0000 7f52bf8297c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:02:17.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:18.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:18.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:18.131 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:02:18.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:02:18.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:18.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:18.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:19.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:19.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:19.643 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:02:19.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:02:19.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:19.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:20.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:21.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:21.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:21.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:02:21.238 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:02:21.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:21.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:22.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:23.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:23.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:23.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:02:23.008 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:02:23.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:23.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:23.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:24.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:24.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:24.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:02:24.604 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:02:24.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:24.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:25.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:26.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:26.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:26.345 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:02:26.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:02:26.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:26.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:26.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:27.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:27.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:27.825 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:02:27.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:02:27.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:27.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:28.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:29.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:29.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:29.494 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:02:29.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:02:29.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:29.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:30.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:31.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:31.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:31.249 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:02:31.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:02:31.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:31.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:31.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:32.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:32.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:32.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:02:32.786 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:02:32.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:32.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:33.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:34.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:34.783 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:02:34.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:34.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:02:34.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:34.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:35.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:36.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:36.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:36.514 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:02:36.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:02:36.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:36.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:37.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:38.686 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:02:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:02:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:39.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:40.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:40.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:40.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:02:40.522 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:02:40.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:40.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:41.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:42.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:42.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:42.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:02:42.241 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:02:42.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:42.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:43.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:44.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:44.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:44.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:02:44.162 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:02:44.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:44.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:45.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:46.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:46.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:46.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:02:46.138 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T08:02:46.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:46.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:46.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:47.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:47.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:47.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:02:47.899 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T08:02:47.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:47.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:49.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:02:50.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:02:50.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:02:50.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:02:50.336 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T08:02:50.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:02:50.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:02:50.771 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3919095685,v1:127.0.0.1:6819/3919095685] [v2:127.0.0.1:6820/3919095685,v1:127.0.0.1:6821/3919095685] exists,up b05d606c-e545-4c38-b3d2-4bbf3083b409 2024-04-25T08:02:50.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:02:50.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:02:50.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:02:50.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:67: TEST_markdown_exceed_maxdown_count: create_rbd_pool 2024-04-25T08:02:50.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T08:02:51.156 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' does not exist 2024-04-25T08:02:51.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T08:02:51.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T08:02:51.731 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' already exists 2024-04-25T08:02:51.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:02:52.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T08:02:56.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:70: TEST_markdown_exceed_maxdown_count: local count=3 2024-04-25T08:02:56.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:71: TEST_markdown_exceed_maxdown_count: local sleeptime=10 2024-04-25T08:02:56.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:72: TEST_markdown_exceed_maxdown_count: local period=300 2024-04-25T08:02:56.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:73: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_count 3' 2024-04-25T08:02:56.661 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T08:02:56.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:74: TEST_markdown_exceed_maxdown_count: ceph tell osd.0 injectargs '--osd_max_markdown_period 300' 2024-04-25T08:02:56.820 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T08:02:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:76: TEST_markdown_exceed_maxdown_count: markdown_N_impl 4 300 10 2024-04-25T08:02:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:39: markdown_N_impl: markdown_times=4 2024-04-25T08:02:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:40: markdown_N_impl: total_time=300 2024-04-25T08:02:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:41: markdown_N_impl: sleeptime=10 2024-04-25T08:02:56.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: seq 1 4 2024-04-25T08:02:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T08:02:56.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T08:02:56.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T08:02:57.401 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T08:02:57.401 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T08:02:57.401 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T08:02:57.401 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:02:57.401 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T08:02:57.402 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T08:02:57.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T08:02:57.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T08:02:57.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T08:02:57.840 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:02:57.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T08:02:57.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T08:02:58.293 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T08:02:58.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T08:03:08.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T08:03:08.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T08:03:08.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T08:03:08.855 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T08:03:08.856 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T08:03:08.856 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T08:03:08.856 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:03:08.856 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T08:03:08.856 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T08:03:08.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T08:03:08.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T08:03:08.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T08:03:09.289 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:03:09.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T08:03:09.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T08:03:09.809 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T08:03:09.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T08:03:19.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T08:03:19.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T08:03:19.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T08:03:20.383 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T08:03:20.383 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T08:03:20.383 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T08:03:20.383 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:03:20.383 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T08:03:20.383 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T08:03:20.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T08:03:20.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T08:03:20.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T08:03:20.883 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:03:20.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T08:03:20.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T08:03:21.403 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T08:03:21.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T08:03:31.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:42: markdown_N_impl: for i in `seq 1 $markdown_times` 2024-04-25T08:03:31.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:45: markdown_N_impl: ceph tell osd.0 get_latest_osdmap 2024-04-25T08:03:31.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:46: markdown_N_impl: ceph osd tree 2024-04-25T08:03:31.977 INFO:tasks.workunit.client.0.smithi134.stdout:ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF 2024-04-25T08:03:31.977 INFO:tasks.workunit.client.0.smithi134.stdout:-1 0.29306 root default 2024-04-25T08:03:31.977 INFO:tasks.workunit.client.0.smithi134.stdout:-3 0.29306 host smithi134 2024-04-25T08:03:31.977 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:03:31.978 INFO:tasks.workunit.client.0.smithi134.stdout: 1 hdd 0.09769 osd.1 up 1.00000 1.00000 2024-04-25T08:03:31.978 INFO:tasks.workunit.client.0.smithi134.stdout: 2 hdd 0.09769 osd.2 up 1.00000 1.00000 2024-04-25T08:03:31.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: ceph osd tree 2024-04-25T08:03:31.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep osd.0 2024-04-25T08:03:31.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:47: markdown_N_impl: grep up 2024-04-25T08:03:32.463 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:03:32.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: unset CEPH_CLI_TEST_DUP_COMMAND 2024-04-25T08:03:32.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:52: markdown_N_impl: ceph osd down 0 2024-04-25T08:03:33.021 INFO:tasks.workunit.client.0.smithi134.stderr:marked down osd.0. 2024-04-25T08:03:33.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:53: markdown_N_impl: sleep 10 2024-04-25T08:03:43.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: ceph osd tree 2024-04-25T08:03:43.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep down 2024-04-25T08:03:43.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:78: TEST_markdown_exceed_maxdown_count: grep osd.0 2024-04-25T08:03:43.474 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-04-25T08:03:43.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:03:43.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:03:43.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:03:43.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:03:43.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:03:43.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:03:43.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:03:43.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:03:43.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:03:43.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:43.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:03:43.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:03:43.588 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:43.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:03:43.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:03:43.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:03:43.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:03:43.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:43.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:03:43.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178330 2024-04-25T08:03:43.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:03:43.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:03:43.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:31: run: for func in $funcs 2024-04-25T08:03:43.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:32: run: setup td/osd-markdown 2024-04-25T08:03:43.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-markdown 2024-04-25T08:03:43.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-markdown 2024-04-25T08:03:43.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:03:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:03:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:03:43.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:03:43.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:03:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:03:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:03:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:03:43.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:03:43.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:03:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:03:43.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:03:43.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:03:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:03:43.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:03:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:43.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:03:43.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:03:43.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:03:43.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:03:43.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:03:43.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:03:43.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:03:43.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:43.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:03:43.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178330 2024-04-25T08:03:43.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:03:43.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:03:43.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-markdown 2024-04-25T08:03:43.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:03:43.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:43.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:03:43.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.178330 2024-04-25T08:03:43.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:03:43.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:03:43.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:03:43.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-markdown 1' TERM HUP INT 2024-04-25T08:03:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:33: run: TEST_osd_stop td/osd-markdown 2024-04-25T08:03:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:132: TEST_osd_stop: local dir=td/osd-markdown 2024-04-25T08:03:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:134: TEST_osd_stop: run_mon td/osd-markdown a 2024-04-25T08:03:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-markdown 2024-04-25T08:03:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:03:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:03:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:03:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-markdown/a 2024-04-25T08:03:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-markdown/a --run-dir=td/osd-markdown 2024-04-25T08:03:44.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:03:44.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:03:44.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:03:44.104 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:03:44.104 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:44.104 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:03:44.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:03:44.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-markdown/a '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --mon-cluster-log-file=td/osd-markdown/log --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:03:44.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:03:44.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:03:44.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:03:44.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:03:44.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:03:44.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:03:44.357 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:03:44.357 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:03:44.357 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:03:44.358 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:03:44.358 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:44.358 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:03:44.358 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178330/ceph-mon.a.asok 2024-04-25T08:03:44.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:03:44.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178330/ceph-mon.a.asok config get fsid 2024-04-25T08:03:44.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:03:44.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:03:44.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:03:44.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.178330/ceph-mon.a.asok 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:03:44.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.178330/ceph-mon.a.asok config get mon_host 2024-04-25T08:03:44.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:135: TEST_osd_stop: run_mgr td/osd-markdown x 2024-04-25T08:03:44.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-markdown 2024-04-25T08:03:44.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:03:44.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:03:44.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:03:44.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-markdown/x 2024-04-25T08:03:44.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:03:45.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:03:45.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:03:45.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:03:45.564 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:03:45.564 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:45.564 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:03:45.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:03:45.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:03:45.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-markdown/x '--log-file=td/osd-markdown/$name.log' '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --run-dir=td/osd-markdown '--pid-file=td/osd-markdown/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:136: TEST_osd_stop: run_osd td/osd-markdown 0 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/0 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/0' 2024-04-25T08:03:45.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/0/journal' 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:03:45.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/0 2024-04-25T08:03:45.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:03:45.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=58e3bed0-dae6-4b9a-8081-1c9db235f54c 2024-04-25T08:03:45.652 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 58e3bed0-dae6-4b9a-8081-1c9db235f54c 2024-04-25T08:03:45.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 58e3bed0-dae6-4b9a-8081-1c9db235f54c' 2024-04-25T08:03:45.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:03:45.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBhDipmOFfeJxAA8u9rnzje9M/xnJ2JFdD9jg== 2024-04-25T08:03:45.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBhDipmOFfeJxAA8u9rnzje9M/xnJ2JFdD9jg=="}' 2024-04-25T08:03:45.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 58e3bed0-dae6-4b9a-8081-1c9db235f54c -i td/osd-markdown/0/new.json 2024-04-25T08:03:46.019 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:03:46.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/0/new.json 2024-04-25T08:03:46.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBhDipmOFfeJxAA8u9rnzje9M/xnJ2JFdD9jg== --osd-uuid 58e3bed0-dae6-4b9a-8081-1c9db235f54c 2024-04-25T08:03:46.064 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:46.062+0000 7f7c202787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:46.103 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:46.102+0000 7f7c202787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:46.105 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:46.102+0000 7f7c202787c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:46.106 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:46.102+0000 7f7c202787c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T08:03:46.106 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:46.102+0000 7f7c202787c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T08:03:46.106 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:46.102+0000 7f7c202787c0 -1 bluestore(td/osd-markdown/0/block) _read_bdev_label failed to open td/osd-markdown/0/block: (2) No such file or directory 2024-04-25T08:03:46.143 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:46.142+0000 7f7c202787c0 -1 bluestore(td/osd-markdown/0) _read_fsid unparsable uuid 2024-04-25T08:03:53.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/0/keyring 2024-04-25T08:03:53.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:03:53.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:03:53.966 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:03:53.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:03:54.543 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:03:54.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:03:54.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:03:54.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:03:54.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/0 --osd-journal=td/osd-markdown/0/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:03:54.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:03:54.572 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:54.570+0000 7f7d693967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:54.574 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:54.570+0000 7f7d693967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:54.575 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:54.574+0000 7f7d693967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:03:54.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:03:54.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:03:54.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:03:54.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:03:54.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:03:54.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:54.955 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:03:54.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:03:54.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:54.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:55.121 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:55.118+0000 7f7d693967c0 -1 Falling back to public interface 2024-04-25T08:03:55.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:56.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:56.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:56.344 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:03:56.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:03:56.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:56.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:56.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:57.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:57.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:57.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:03:57.745 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:03:57.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:57.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:57.965 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:03:57.962+0000 7f7d693967c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:03:58.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:03:59.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:03:59.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:03:59.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:03:59.277 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:03:59.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:03:59.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:03:59.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:00.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:00.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:00.739 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:04:00.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:04:00.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:00.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:01.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:02.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:02.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:02.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:04:02.362 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:04:02.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:02.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:02.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:03.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:03.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:03.805 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:04:03.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:04:03.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:03.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:04.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:05.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:05.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:05.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:04:05.502 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:04:05.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:05.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:06.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:04:07.152 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:04:07.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:07.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:07.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:08.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:08.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:08.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:04:08.641 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:04:08.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:08.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:09.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:10.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:10.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:10.264 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:04:10.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:04:10.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:10.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:10.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:11.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:11.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:11.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:04:11.871 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:04:11.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:11.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:12.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:13.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:13.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:13.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:04:13.484 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:04:13.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:13.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:14.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:15.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:15.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:15.128 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:04:15.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:04:15.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:15.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:15.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:16.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:16.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:16.608 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:04:16.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:04:16.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:16.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:17.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:18.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:18.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:18.633 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:04:18.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:04:18.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:18.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:20.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:20.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:20.539 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:04:20.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:04:20.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:20.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:21.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:22.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:22.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:22.899 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:04:22.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:04:22.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:22.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:24.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:25.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:25.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:25.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:04:25.112 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:04:25.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:25.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:26.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:27.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:27.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:27.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:04:27.504 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T08:04:27.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:27.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:28.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:29.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:29.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:29.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:04:29.331 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T08:04:29.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:29.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:30.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:31.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:31.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:31.016 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T08:04:31.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:04:31.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:31.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:04:31.409 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4178916579,v1:127.0.0.1:6803/4178916579] [v2:127.0.0.1:6804/4178916579,v1:127.0.0.1:6805/4178916579] exists,up 58e3bed0-dae6-4b9a-8081-1c9db235f54c 2024-04-25T08:04:31.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:04:31.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:04:31.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:04:31.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:137: TEST_osd_stop: run_osd td/osd-markdown 1 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/1 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/1' 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/1/journal' 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:04:31.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T08:04:31.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/1 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=170865f1-d69c-4605-a222-4da25177f70b 2024-04-25T08:04:31.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 170865f1-d69c-4605-a222-4da25177f70b' 2024-04-25T08:04:31.417 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 170865f1-d69c-4605-a222-4da25177f70b 2024-04-25T08:04:31.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:04:31.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCPDipmP73jGRAATVkreKqv+b/eoRwoZneY+w== 2024-04-25T08:04:31.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCPDipmP73jGRAATVkreKqv+b/eoRwoZneY+w=="}' 2024-04-25T08:04:31.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 170865f1-d69c-4605-a222-4da25177f70b -i td/osd-markdown/1/new.json 2024-04-25T08:04:32.016 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:04:32.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/1/new.json 2024-04-25T08:04:32.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCPDipmP73jGRAATVkreKqv+b/eoRwoZneY+w== --osd-uuid 170865f1-d69c-4605-a222-4da25177f70b 2024-04-25T08:04:32.060 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:32.061+0000 7ff9b0ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:32.063 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:32.061+0000 7ff9b0ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:32.064 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:32.065+0000 7ff9b0ee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:32.064 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:32.065+0000 7ff9b0ee47c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T08:04:32.064 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:32.065+0000 7ff9b0ee47c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T08:04:32.064 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:32.065+0000 7ff9b0ee47c0 -1 bluestore(td/osd-markdown/1/block) _read_bdev_label failed to open td/osd-markdown/1/block: (2) No such file or directory 2024-04-25T08:04:32.250 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:32.249+0000 7ff9b0ee47c0 -1 bluestore(td/osd-markdown/1) _read_fsid unparsable uuid 2024-04-25T08:04:40.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/1/keyring 2024-04-25T08:04:40.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:04:40.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:04:40.272 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:04:40.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:04:40.832 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:04:40.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:04:40.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/1 --osd-journal=td/osd-markdown/1/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:04:40.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:04:40.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:04:40.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:04:40.855 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:40.853+0000 7fc5a42967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:40.857 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:40.857+0000 7fc5a42967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:40.858 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:40.857+0000 7fc5a42967c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:04:41.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:04:41.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:04:41.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:04:41.231 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:04:41.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:04:41.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:04:41.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:41.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:04:41.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:41.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:41.408 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:41.409+0000 7fc5a42967c0 -1 Falling back to public interface 2024-04-25T08:04:41.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:42.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:42.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:42.671 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:04:42.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:04:42.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:42.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:43.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:44.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:44.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:44.147 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:04:44.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:04:44.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:44.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:04:44.249+0000 7fc5a42967c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:04:44.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:45.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:45.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:45.629 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:04:45.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:04:45.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:45.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:46.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:47.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:47.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:47.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:04:47.145 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:04:47.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:47.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:47.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:48.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:48.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:48.678 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:04:48.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:04:48.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:48.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:49.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:50.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:50.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:50.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:04:50.276 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:04:50.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:50.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:50.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:51.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:51.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:51.928 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:04:51.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:04:51.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:51.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:52.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:53.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:53.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:53.610 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:04:53.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:04:53.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:53.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:54.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:55.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:55.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:55.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:04:55.096 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:04:55.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:55.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:55.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:56.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:56.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:56.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:04:56.796 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:04:56.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:56.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:57.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:04:58.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:04:58.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:04:58.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:04:58.383 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:04:58.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:04:58.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:04:59.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:00.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:00.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:00.012 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:05:00.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:05:00.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:00.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:00.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:01.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:01.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:01.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:05:01.653 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:05:01.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:01.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:02.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:03.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:03.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:03.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:05:03.113 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:05:03.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:03.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:03.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:04.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:04.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:04.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:05:04.792 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:05:04.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:04.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:05.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:06.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:06.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:06.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:05:06.872 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:05:06.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:06.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:08.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:09.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:09.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:09.062 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:05:09.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:05:09.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:09.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:10.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:11.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:11.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:11.141 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:05:11.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:05:11.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:11.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:12.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:13.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:13.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:13.488 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T08:05:13.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:05:13.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:13.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:14.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:15.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:15.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:15.344 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T08:05:15.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:05:15.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:15.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:16.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:17.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:17.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:17.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:05:17.080 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T08:05:17.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:17.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:05:17.620 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1478114927,v1:127.0.0.1:6811/1478114927] [v2:127.0.0.1:6812/1478114927,v1:127.0.0.1:6813/1478114927] exists,up 170865f1-d69c-4605-a222-4da25177f70b 2024-04-25T08:05:17.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:05:17.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:05:17.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:05:17.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:138: TEST_osd_stop: run_osd td/osd-markdown 2 2024-04-25T08:05:17.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-markdown 2024-04-25T08:05:17.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-markdown/2 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 ' 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-markdown/2' 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-markdown/2/journal' 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-markdown' 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:05:17.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:05:17.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:05:17.624 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:05:17.624 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:05:17.624 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:05:17.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-markdown/$name.log' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-markdown/$name.pid' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:05:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:05:17.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-markdown/2 2024-04-25T08:05:17.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:05:17.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0c69818f-1249-402d-9377-157d08610e61 2024-04-25T08:05:17.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0c69818f-1249-402d-9377-157d08610e61' 2024-04-25T08:05:17.627 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 0c69818f-1249-402d-9377-157d08610e61 2024-04-25T08:05:17.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:05:17.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9DipmVY+rJhAAIzuh03MosYZSnFx4ptorQg== 2024-04-25T08:05:17.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9DipmVY+rJhAAIzuh03MosYZSnFx4ptorQg=="}' 2024-04-25T08:05:17.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0c69818f-1249-402d-9377-157d08610e61 -i td/osd-markdown/2/new.json 2024-04-25T08:05:18.242 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:05:18.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-markdown/2/new.json 2024-04-25T08:05:18.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9DipmVY+rJhAAIzuh03MosYZSnFx4ptorQg== --osd-uuid 0c69818f-1249-402d-9377-157d08610e61 2024-04-25T08:05:18.292 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:18.292+0000 7fb536ff27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:18.294 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:18.292+0000 7fb536ff27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:18.296 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:18.296+0000 7fb536ff27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:18.296 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:18.296+0000 7fb536ff27c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T08:05:18.296 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:18.296+0000 7fb536ff27c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T08:05:18.297 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:18.296+0000 7fb536ff27c0 -1 bluestore(td/osd-markdown/2/block) _read_bdev_label failed to open td/osd-markdown/2/block: (2) No such file or directory 2024-04-25T08:05:18.417 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:18.416+0000 7fb536ff27c0 -1 bluestore(td/osd-markdown/2) _read_fsid unparsable uuid 2024-04-25T08:05:26.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-markdown/2/keyring 2024-04-25T08:05:26.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:05:26.545 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T08:05:26.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:05:26.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-markdown/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:05:27.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:05:27.136 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T08:05:27.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=17fc6dd9-c396-461d-ae21-656c39a7dca3 --auth-supported=none --mon-host=127.0.0.1:7108 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-markdown/2 --osd-journal=td/osd-markdown/2/journal --chdir= --run-dir=td/osd-markdown '--admin-socket=/tmp/ceph-asok.178330/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-markdown/$name.log' '--pid-file=td/osd-markdown/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:05:27.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:05:27.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:05:27.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:05:27.165 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:27.164+0000 7fb29fa0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:27.166 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:27.164+0000 7fb29fa0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:27.168 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:27.168+0000 7fb29fa0d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:05:27.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:05:27.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:05:27.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:05:27.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:05:27.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:05:27.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:27.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:05:27.547 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:05:27.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:27.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:27.709 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:27.708+0000 7fb29fa0d7c0 -1 Falling back to public interface 2024-04-25T08:05:27.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:28.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:28.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:28.942 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:05:28.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:05:28.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:28.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:29.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:30.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:30.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:30.366 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:05:30.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:05:30.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:30.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:30.641 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:05:30.640+0000 7fb29fa0d7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:05:30.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:31.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:31.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:31.775 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:05:31.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:05:31.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:31.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:32.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:33.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:33.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:33.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:05:33.281 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:05:33.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:33.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:33.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:34.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:34.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:34.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:05:34.922 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:05:34.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:34.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:35.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:36.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:36.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:36.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:05:36.465 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:05:36.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:36.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:37.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:38.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:38.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:38.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:05:38.039 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:05:38.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:38.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:38.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:39.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:39.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:39.527 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:05:39.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:05:39.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:39.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:40.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:41.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:41.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:41.208 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:05:41.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:05:41.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:41.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:41.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:42.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:42.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:42.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:05:42.887 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:05:42.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:42.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:44.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:44.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:44.614 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:05:44.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:05:44.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:44.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:45.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:46.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:46.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:46.249 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:05:46.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:05:46.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:46.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:46.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:47.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:47.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:47.872 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:05:47.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:05:47.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:47.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:48.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:49.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:49.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:49.368 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:05:49.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:05:49.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:49.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:50.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:51.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:51.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:51.047 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:05:51.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:05:51.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:51.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:52.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:53.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:53.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:53.037 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:05:53.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:05:53.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:53.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:53.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:54.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:54.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:54.813 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:05:54.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:05:54.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:54.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:56.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:57.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:57.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:57.185 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:05:57.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:05:57.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:57.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:57.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:05:58.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:05:58.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:05:58.670 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T08:05:58.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:05:58.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:05:58.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:05:59.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:00.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:00.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:00.618 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T08:06:00.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:06:00.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:00.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:06:01.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:02.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:02.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:02.964 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T08:06:02.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:06:02.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:02.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:06:03.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:04.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:04.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:04.574 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T08:06:04.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-04-25T08:06:04.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:04.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:06:04.990 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/58260663,v1:127.0.0.1:6819/58260663] [v2:127.0.0.1:6820/58260663,v1:127.0.0.1:6821/58260663] exists,up 0c69818f-1249-402d-9377-157d08610e61 2024-04-25T08:06:04.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:06:04.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:06:04.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:06:04.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: cat td/osd-markdown/osd.0.pid 2024-04-25T08:06:04.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:139: TEST_osd_stop: osd_0_pid=198489 2024-04-25T08:06:04.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:140: TEST_osd_stop: ps -p 198489 2024-04-25T08:06:05.000 INFO:tasks.workunit.client.0.smithi134.stdout: PID TTY TIME CMD 2024-04-25T08:06:05.000 INFO:tasks.workunit.client.0.smithi134.stdout: 198489 ? 00:00:01 ceph-osd 2024-04-25T08:06:05.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: ceph osd tree 2024-04-25T08:06:05.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep osd.0 2024-04-25T08:06:05.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:142: TEST_osd_stop: grep up 2024-04-25T08:06:05.446 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 up 1.00000 1.00000 2024-04-25T08:06:05.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:143: TEST_osd_stop: ceph osd stop osd.0 2024-04-25T08:06:06.172 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already stopped. 2024-04-25T08:06:06.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:144: TEST_osd_stop: sleep 15 2024-04-25T08:06:21.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: ceph osd tree 2024-04-25T08:06:21.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep down 2024-04-25T08:06:21.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:145: TEST_osd_stop: grep osd.0 2024-04-25T08:06:21.606 INFO:tasks.workunit.client.0.smithi134.stdout: 0 hdd 0.09769 osd.0 down 1.00000 1.00000 2024-04-25T08:06:21.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:146: TEST_osd_stop: ps -p 198489 2024-04-25T08:06:21.615 INFO:tasks.workunit.client.0.smithi134.stdout: PID TTY TIME CMD 2024-04-25T08:06:21.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-markdown.sh:34: run: teardown td/osd-markdown 2024-04-25T08:06:21.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:06:21.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:06:21.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:06:21.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:06:21.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:06:21.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:06:21.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:06:21.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:06:21.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:06:21.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:06:21.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:06:21.726 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:06:21.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:06:21.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:06:21.727 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:06:21.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:21.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:06:21.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:06:21.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:21.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:06:21.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:06:21.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:06:21.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:06:21.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:21.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:06:21.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178330 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-markdown 0 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-markdown 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-markdown KILL 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:06:21.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:06:21.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:06:21.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:06:21.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:06:21.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:06:21.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:06:21.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:06:21.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:06:21.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:21.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:06:21.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:06:21.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:21.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:06:21.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:06:21.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-markdown 2024-04-25T08:06:21.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:06:21.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:21.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.178330 2024-04-25T08:06:21.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.178330 2024-04-25T08:06:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:06:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:06:21.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:06:21.756 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:06:21.756 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:06:21.808 INFO:tasks.workunit:Running workunit osd/osd-reactivate.sh... 2024-04-25T08:06:21.808 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-reactivate.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh 2024-04-25T08:06:21.861 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:06:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:06:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:06:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:06:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:06:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:06:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:06:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:06:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-reactivate 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:19: run: local dir=td/osd-reactivate 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:20: run: shift 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: export CEPH_MON=127.0.0.1:7122 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:22: run: CEPH_MON=127.0.0.1:7122 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:23: run: export CEPH_ARGS 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: uuidgen 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:24: run: CEPH_ARGS+='--fsid=3998da67-5499-4bb2-94f8-6c0b6918d981 --auth-supported=none ' 2024-04-25T08:06:21.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:25: run: CEPH_ARGS+='--mon-host=127.0.0.1:7122 ' 2024-04-25T08:06:21.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: set 2024-04-25T08:06:21.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:06:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:27: run: local funcs=TEST_reactivate 2024-04-25T08:06:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:28: run: for func in $funcs 2024-04-25T08:06:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:29: run: setup td/osd-reactivate 2024-04-25T08:06:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reactivate 2024-04-25T08:06:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reactivate 2024-04-25T08:06:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T08:06:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:06:21.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T08:06:21.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:06:21.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:06:21.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:06:21.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:06:21.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:06:21.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:06:21.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:06:21.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:06:21.872 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:06:21.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:06:21.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:06:21.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:06:21.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:21.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:06:21.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:06:21.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:06:21.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:06:21.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:06:21.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T08:06:21.878 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:06:21.878 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:21.878 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:06:21.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.203329 2024-04-25T08:06:21.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:06:21.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:06:21.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reactivate 2024-04-25T08:06:21.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:06:21.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:21.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:06:21.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.203329 2024-04-25T08:06:21.882 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:06:21.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:06:21.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:06:21.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:06:21.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reactivate 1' TERM HUP INT 2024-04-25T08:06:21.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:30: run: TEST_reactivate td/osd-reactivate 2024-04-25T08:06:21.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:36: TEST_reactivate: local dir=td/osd-reactivate 2024-04-25T08:06:21.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:38: TEST_reactivate: run_mon td/osd-reactivate a 2024-04-25T08:06:21.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reactivate 2024-04-25T08:06:21.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:06:21.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:06:21.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:06:21.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reactivate/a 2024-04-25T08:06:21.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reactivate/a --run-dir=td/osd-reactivate 2024-04-25T08:06:22.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:06:22.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:06:22.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:06:22.399 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:06:22.399 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:22.400 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:06:22.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203329/$cluster-$name.asok' 2024-04-25T08:06:22.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reactivate/a '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.203329/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reactivate/log --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:06:22.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:06:22.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:06:22.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:06:22.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:06:22.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:06:22.652 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:06:22.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:06:22.652 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:06:22.652 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:06:22.652 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:06:22.652 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:22.652 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:06:22.653 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203329/ceph-mon.a.asok 2024-04-25T08:06:22.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:06:22.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.203329/ceph-mon.a.asok config get fsid 2024-04-25T08:06:22.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:06:22.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:06:22.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:06:22.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:06:22.770 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:06:22.770 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:06:22.770 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:06:22.770 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:06:22.770 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:06:22.771 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:22.771 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:06:22.771 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.203329/ceph-mon.a.asok 2024-04-25T08:06:22.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:06:22.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.203329/ceph-mon.a.asok config get mon_host 2024-04-25T08:06:22.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:39: TEST_reactivate: run_mgr td/osd-reactivate x 2024-04-25T08:06:22.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reactivate 2024-04-25T08:06:22.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:06:22.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:06:22.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:06:22.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reactivate/x 2024-04-25T08:06:22.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:06:23.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:06:23.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:06:23.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:06:23.837 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:06:23.837 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:23.837 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:06:23.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203329/$cluster-$name.asok' 2024-04-25T08:06:23.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:06:23.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reactivate/x '--log-file=td/osd-reactivate/$name.log' '--admin-socket=/tmp/ceph-asok.203329/$cluster-$name.asok' --run-dir=td/osd-reactivate '--pid-file=td/osd-reactivate/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:40: TEST_reactivate: run_osd td/osd-reactivate 0 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reactivate 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reactivate/0 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=3998da67-5499-4bb2-94f8-6c0b6918d981 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:06:23.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203329/$cluster-$name.asok' 2024-04-25T08:06:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.203329/$cluster-$name.asok' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reactivate/0 2024-04-25T08:06:23.878 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:06:23.879 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 5974f602-3d7e-4c2f-9519-6a75170195f1 2024-04-25T08:06:23.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5974f602-3d7e-4c2f-9519-6a75170195f1 2024-04-25T08:06:23.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5974f602-3d7e-4c2f-9519-6a75170195f1' 2024-04-25T08:06:23.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:06:23.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/DipmvmdRNRAAhMWR4p7TqyqnFbT3vzhMlw== 2024-04-25T08:06:23.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/DipmvmdRNRAAhMWR4p7TqyqnFbT3vzhMlw=="}' 2024-04-25T08:06:23.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5974f602-3d7e-4c2f-9519-6a75170195f1 -i td/osd-reactivate/0/new.json 2024-04-25T08:06:24.281 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:06:24.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reactivate/0/new.json 2024-04-25T08:06:24.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=3998da67-5499-4bb2-94f8-6c0b6918d981 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.203329/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD/DipmvmdRNRAAhMWR4p7TqyqnFbT3vzhMlw== --osd-uuid 5974f602-3d7e-4c2f-9519-6a75170195f1 2024-04-25T08:06:24.330 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:24.327+0000 7f5d868c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:24.374 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:24.371+0000 7f5d868c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:24.375 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:24.371+0000 7f5d868c47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:24.376 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:24.375+0000 7f5d868c47c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T08:06:24.376 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:24.375+0000 7f5d868c47c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T08:06:24.376 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:24.375+0000 7f5d868c47c0 -1 bluestore(td/osd-reactivate/0/block) _read_bdev_label failed to open td/osd-reactivate/0/block: (2) No such file or directory 2024-04-25T08:06:24.414 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:24.411+0000 7f5d868c47c0 -1 bluestore(td/osd-reactivate/0) _read_fsid unparsable uuid 2024-04-25T08:06:32.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reactivate/0/keyring 2024-04-25T08:06:32.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:06:32.254 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:06:32.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:06:32.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reactivate/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:06:32.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:06:32.817 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:06:32.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:06:32.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=3998da67-5499-4bb2-94f8-6c0b6918d981 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.203329/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:06:32.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:06:32.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:06:32.846 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:32.843+0000 7f369d5677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:32.848 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:32.847+0000 7f369d5677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:32.849 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:32.847+0000 7f369d5677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:06:33.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:06:33.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:06:33.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:06:33.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:06:33.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:06:33.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:33.204 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:06:33.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:06:33.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:33.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:33.398 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:33.394+0000 7f369d5677c0 -1 Falling back to public interface 2024-04-25T08:06:33.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:34.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:34.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:34.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:06:34.624 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:06:34.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:34.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:35.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:36.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:36.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:36.026 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:06:36.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:06:36.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:36.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:36.276 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:06:36.274+0000 7f369d5677c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:06:36.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:37.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:37.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:37.443 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:06:37.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:06:37.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:37.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:38.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:39.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:39.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:39.030 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:06:39.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:06:39.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:39.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:39.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:40.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:40.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:40.617 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:06:40.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:06:40.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:40.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:41.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:42.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:42.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:42.088 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:06:42.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:06:42.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:42.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:42.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:43.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:43.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:43.725 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:06:43.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:06:43.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:43.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:44.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:45.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:45.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:45.393 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:06:45.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:06:45.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:45.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:45.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:46.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:46.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:46.894 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:06:46.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:06:46.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:46.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:48.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:48.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:48.578 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:06:48.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:06:48.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:48.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:49.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:50.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:50.283 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:06:50.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:50.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:06:50.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:50.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:50.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:51.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:51.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:51.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:06:51.810 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:06:51.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:51.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:52.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:53.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:53.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:53.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:06:53.411 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:06:53.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:53.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:53.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:54.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:54.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:54.857 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:06:54.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:06:54.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:54.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:55.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:56.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:56.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:56.829 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:06:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:06:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:57.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:06:58.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:06:58.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:06:58.635 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:06:58.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:06:58.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:06:58.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:06:59.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:00.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:00.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:00.399 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:07:00.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:07:00.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:00.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:01.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:02.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:02.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:02.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:07:02.577 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:07:02.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:02.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:03.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:04.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:04.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:04.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:07:04.576 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T08:07:04.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:04.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:05.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:06.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:06.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:06.627 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T08:07:06.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-04-25T08:07:06.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:06.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:07.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:08.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:08.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:08.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-04-25T08:07:08.831 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T08:07:08.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:08.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:09.246 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1859296137,v1:127.0.0.1:6803/1859296137] [v2:127.0.0.1:6804/1859296137,v1:127.0.0.1:6805/1859296137] exists,up 5974f602-3d7e-4c2f-9519-6a75170195f1 2024-04-25T08:07:09.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:07:09.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:07:09.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:07:09.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:42: TEST_reactivate: kill_daemons td/osd-reactivate TERM osd 2024-04-25T08:07:09.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:07:09.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:07:09.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:07:09.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:07:09.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:07:09.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:07:09.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:44: TEST_reactivate: ready_path=td/osd-reactivate/0/ready 2024-04-25T08:07:09.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:45: TEST_reactivate: activate_path=td/osd-reactivate/0/active 2024-04-25T08:07:09.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:47: TEST_reactivate: rm -rf td/osd-reactivate/0/ready td/osd-reactivate/0/active 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:48: TEST_reactivate: activate_osd td/osd-reactivate 0 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-reactivate 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-reactivate/0 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=3998da67-5499-4bb2-94f8-6c0b6918d981 --auth-supported=none --mon-host=127.0.0.1:7122 ' 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-reactivate/0' 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-reactivate/0/journal' 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:07:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-reactivate' 2024-04-25T08:07:09.559 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:07:09.559 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:07:09.559 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:07:09.559 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:07:09.559 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:09.559 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:07:09.559 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.203329/$cluster-$name.asok' 2024-04-25T08:07:09.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.203329/$cluster-$name.asok' 2024-04-25T08:07:09.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-reactivate/$name.log' 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-reactivate/$name.pid' 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-reactivate/0 2024-04-25T08:07:09.560 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:07:09.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T08:07:09.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=3998da67-5499-4bb2-94f8-6c0b6918d981 --auth-supported=none --mon-host=127.0.0.1:7122 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reactivate/0 --osd-journal=td/osd-reactivate/0/journal --chdir= --run-dir=td/osd-reactivate '--admin-socket=/tmp/ceph-asok.203329/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-reactivate/$name.log' '--pid-file=td/osd-reactivate/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:07:09.561 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-reactivate/0/whoami 2024-04-25T08:07:09.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T08:07:09.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:07:09.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:07:09.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:07:09.589 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:09.586+0000 7fad743cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:09.595 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:09.594+0000 7fad743cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:09.597 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:09.594+0000 7fad743cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:10.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T08:07:10.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:07:10.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:07:10.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:07:10.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:07:10.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:10.087 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:07:10.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:07:10.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:10.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:10.165 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:10.162+0000 7fad743cc7c0 -1 Falling back to public interface 2024-04-25T08:07:10.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:11.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:11.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:11.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:07:11.548 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:07:11.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:11.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:11.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:12.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:12.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:12.964 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:07:12.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:07:12.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:12.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:13.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:13.903 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:13.902+0000 7fad743cc7c0 -1 osd.0 5 log_to_monitors true 2024-04-25T08:07:14.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:14.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:14.538 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:07:14.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:07:14.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:14.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:14.980 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6802/1280505568,v1:127.0.0.1:6803/1280505568] [v2:127.0.0.1:6804/1280505568,v1:127.0.0.1:6805/1280505568] exists,up 5974f602-3d7e-4c2f-9519-6a75170195f1 2024-04-25T08:07:14.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reactivate.sh:31: run: teardown td/osd-reactivate 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:07:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:07:15.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:07:15.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:07:15.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:07:15.292 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:07:15.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:07:15.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:07:15.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:07:15.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:15.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:07:15.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:07:15.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:15.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:07:15.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:07:15.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T08:07:15.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:07:15.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:15.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:07:15.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.203329 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-reactivate 0 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reactivate 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reactivate KILL 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:07:15.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:07:15.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:07:15.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:07:15.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:07:15.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:07:15.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:07:15.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:07:15.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:07:15.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:07:15.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:07:15.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:15.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:07:15.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:07:15.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:15.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:07:15.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:07:15.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reactivate 2024-04-25T08:07:15.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:07:15.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:15.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.203329 2024-04-25T08:07:15.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.203329 2024-04-25T08:07:15.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:07:15.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:07:15.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:07:15.321 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:07:15.321 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:07:15.376 INFO:tasks.workunit:Running workunit osd/osd-recovery-prio.sh... 2024-04-25T08:07:15.376 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-recovery-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh 2024-04-25T08:07:15.430 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:07:15.438 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:07:15.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:07:15.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:07:15.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:07:15.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:07:15.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:07:15.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:07:15.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:07:15.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-prio 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:21: run: local dir=td/osd-recovery-prio 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:22: run: shift 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:26: run: export CEPH_ARGS 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: uuidgen 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:27: run: CEPH_ARGS+='--fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none ' 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:31: run: CEPH_ARGS+='--osd-op-queue=wpq ' 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: export objects=200 2024-04-25T08:07:15.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:32: run: objects=200 2024-04-25T08:07:15.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: export poolprefix=test 2024-04-25T08:07:15.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:33: run: poolprefix=test 2024-04-25T08:07:15.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: export FORCE_PRIO=255 2024-04-25T08:07:15.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:34: run: FORCE_PRIO=255 2024-04-25T08:07:15.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: export NORMAL_PRIO=190 2024-04-25T08:07:15.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:35: run: NORMAL_PRIO=190 2024-04-25T08:07:15.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: set 2024-04-25T08:07:15.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:37: run: local 'funcs=TEST_recovery_pool_priority 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_recovery_priority' 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:07:15.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:07:15.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:07:15.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:07:15.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:07:15.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:07:15.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:07:15.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:07:15.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:07:15.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:07:15.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:07:15.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:15.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:07:15.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:07:15.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:07:15.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:07:15.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:07:15.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:07:15.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:07:15.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:15.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:07:15.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.206040 2024-04-25T08:07:15.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:07:15.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:07:15.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-04-25T08:07:15.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:07:15.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:15.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:07:15.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.206040 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_pool_priority td/osd-recovery-prio 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:326: TEST_recovery_pool_priority: local dir=td/osd-recovery-prio 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:327: TEST_recovery_pool_priority: local pools=3 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:328: TEST_recovery_pool_priority: local OSDS=2 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:330: TEST_recovery_pool_priority: run_mon td/osd-recovery-prio a 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:07:15.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:07:15.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:07:15.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-04-25T08:07:15.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-04-25T08:07:15.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:07:15.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:07:15.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:07:15.976 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:07:15.976 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:15.976 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:07:15.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:07:15.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:07:16.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:07:16.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:07:16.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:07:16.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:07:16.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:07:16.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:07:16.230 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:07:16.230 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:07:16.231 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:07:16.231 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:07:16.231 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:16.231 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:07:16.231 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-mon.a.asok 2024-04-25T08:07:16.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:07:16.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.206040/ceph-mon.a.asok config get fsid 2024-04-25T08:07:16.347 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:07:16.347 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:07:16.347 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:07:16.347 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:07:16.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:07:16.348 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:07:16.348 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:07:16.348 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:07:16.349 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:07:16.349 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:16.349 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:07:16.349 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-mon.a.asok 2024-04-25T08:07:16.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:07:16.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.206040/ceph-mon.a.asok config get mon_host 2024-04-25T08:07:16.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:331: TEST_recovery_pool_priority: run_mgr td/osd-recovery-prio x 2024-04-25T08:07:16.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-04-25T08:07:16.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:07:16.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:07:16.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:07:16.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-04-25T08:07:16.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:07:17.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:07:17.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:07:17.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:07:17.424 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:07:17.424 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:17.424 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:07:17.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:07:17.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:07:17.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:07:17.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:332: TEST_recovery_pool_priority: export CEPH_ARGS 2024-04-25T08:07:17.527 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: expr 2 - 1 2024-04-25T08:07:17.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: seq 0 1 2024-04-25T08:07:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:07:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 0 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:07:17.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:07:17.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:07:17.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:07:17.533 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:07:17.533 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:17.533 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:07:17.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:07:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-04-25T08:07:17.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:07:17.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2fac8afb-ee38-44ea-8c16-84d66abe9fa5 2024-04-25T08:07:17.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2fac8afb-ee38-44ea-8c16-84d66abe9fa5' 2024-04-25T08:07:17.536 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 2fac8afb-ee38-44ea-8c16-84d66abe9fa5 2024-04-25T08:07:17.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:07:17.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1DypmNlE1IRAAaDw1RkZ37Yv84qKPMDMWuw== 2024-04-25T08:07:17.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1DypmNlE1IRAAaDw1RkZ37Yv84qKPMDMWuw=="}' 2024-04-25T08:07:17.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2fac8afb-ee38-44ea-8c16-84d66abe9fa5 -i td/osd-recovery-prio/0/new.json 2024-04-25T08:07:17.906 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:07:17.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-04-25T08:07:17.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1DypmNlE1IRAAaDw1RkZ37Yv84qKPMDMWuw== --osd-uuid 2fac8afb-ee38-44ea-8c16-84d66abe9fa5 2024-04-25T08:07:17.956 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:17.957+0000 7f4efc9dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:17.992 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:17.993+0000 7f4efc9dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:17.993 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:17.993+0000 7f4efc9dc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:17.994 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:17.993+0000 7f4efc9dc7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:07:17.994 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:17.993+0000 7f4efc9dc7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:07:17.994 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:17.993+0000 7f4efc9dc7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:07:18.023 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:18.021+0000 7f4efc9dc7c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-04-25T08:07:26.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-04-25T08:07:26.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:07:26.012 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:07:26.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:07:26.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:07:26.582 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:07:26.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:07:26.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:07:26.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:07:26.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:07:26.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:07:26.617 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:26.617+0000 7f4cdc3317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:26.619 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:26.621+0000 7f4cdc3317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:26.620 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:26.621+0000 7f4cdc3317c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:27.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:07:27.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:07:27.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:07:27.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:07:27.005 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:07:27.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:07:27.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:27.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:07:27.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:27.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:27.201 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:27.201+0000 7f4cdc3317c0 -1 Falling back to public interface 2024-04-25T08:07:27.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:28.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:28.414 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:07:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:07:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:28.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:29.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:29.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:29.835 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:07:29.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:07:29.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:29.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:30.054 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:30.053+0000 7f4cdc3317c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:07:30.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:31.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:31.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:31.307 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:07:31.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:07:31.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:31.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:07:31.757 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1300051100,v1:127.0.0.1:6803/1300051100] [v2:127.0.0.1:6804/1300051100,v1:127.0.0.1:6805/1300051100] exists,up 2fac8afb-ee38-44ea-8c16-84d66abe9fa5 2024-04-25T08:07:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:07:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:07:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:07:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:334: TEST_recovery_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:07:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:336: TEST_recovery_pool_priority: run_osd td/osd-recovery-prio 1 2024-04-25T08:07:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:07:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:07:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:07:31.760 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:07:31.761 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:07:31.761 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:07:31.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:07:31.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:07:31.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:07:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3745113d-82e1-498a-949e-ed2a4168a408 2024-04-25T08:07:31.764 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 3745113d-82e1-498a-949e-ed2a4168a408 2024-04-25T08:07:31.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 3745113d-82e1-498a-949e-ed2a4168a408' 2024-04-25T08:07:31.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:07:31.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDDypmAqvsLhAA+2Y5bbQIqrSsqg3tDhVysA== 2024-04-25T08:07:31.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDDypmAqvsLhAA+2Y5bbQIqrSsqg3tDhVysA=="}' 2024-04-25T08:07:31.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3745113d-82e1-498a-949e-ed2a4168a408 -i td/osd-recovery-prio/1/new.json 2024-04-25T08:07:32.295 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:07:32.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-04-25T08:07:32.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDDypmAqvsLhAA+2Y5bbQIqrSsqg3tDhVysA== --osd-uuid 3745113d-82e1-498a-949e-ed2a4168a408 2024-04-25T08:07:32.347 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:32.349+0000 7f4d50e877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:32.349 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:32.349+0000 7f4d50e877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:32.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:32.349+0000 7f4d50e877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:32.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:32.349+0000 7f4d50e877c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:07:32.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:32.353+0000 7f4d50e877c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:07:32.351 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:32.353+0000 7f4d50e877c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:07:32.419 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:32.421+0000 7f4d50e877c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-04-25T08:07:40.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-04-25T08:07:40.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:07:40.542 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:07:40.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:07:40.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:07:41.270 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:07:41.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:07:41.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:07:41.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:07:41.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:07:41.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:07:41.304 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:41.305+0000 7f5ae80fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:41.306 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:41.305+0000 7f5ae80fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:41.307 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:41.309+0000 7f5ae80fc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:07:41.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:07:41.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:07:41.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:07:41.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:07:41.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:07:41.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:41.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:07:41.691 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:07:41.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:41.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:07:41.877 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:41.877+0000 7f5ae80fc7c0 -1 Falling back to public interface 2024-04-25T08:07:42.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:43.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:43.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:43.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:07:43.117 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:07:43.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:43.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:07:43.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:44.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:44.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:44.527 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:07:44.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:07:44.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:44.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:07:44.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:07:45.005 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:07:45.005+0000 7f5ae80fc7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:07:45.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:07:45.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:07:45.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:07:45.969 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:07:45.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:07:45.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:07:46.523 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3453618695,v1:127.0.0.1:6811/3453618695] [v2:127.0.0.1:6812/3453618695,v1:127.0.0.1:6813/3453618695] exists,up 3745113d-82e1-498a-949e-ed2a4168a408 2024-04-25T08:07:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:07:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:07:46.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:07:46.523 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: seq 1 3 2024-04-25T08:07:46.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:46.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test1 1 1 2024-04-25T08:07:46.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:07:47.142 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test1' already exists 2024-04-25T08:07:47.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:48.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-04-25T08:07:49.876 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 2 2024-04-25T08:07:49.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:49.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test2 1 1 2024-04-25T08:07:49.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-04-25T08:07:50.688 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test2' already exists 2024-04-25T08:07:50.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:51.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-04-25T08:07:52.686 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 2 size to 2 2024-04-25T08:07:52.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:339: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:07:52.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:341: TEST_recovery_pool_priority: create_pool test3 1 1 2024-04-25T08:07:52.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-04-25T08:07:53.606 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test3' already exists 2024-04-25T08:07:53.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:07:54.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:342: TEST_recovery_pool_priority: ceph osd pool set test3 size 2 2024-04-25T08:07:55.857 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 3 size to 2 2024-04-25T08:07:55.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:344: TEST_recovery_pool_priority: sleep 5 2024-04-25T08:08:00.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:346: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T08:08:00.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:08:00.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:08:00.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:08:00.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:08:00.886 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:08:00.886 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:08:00.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:08:00.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:08:00.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:08:01.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:08:01.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:08:01.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:08:01.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:08:01.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:08:01.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:08:01.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:08:01.439 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T08:08:01.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:08:01.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:08:01.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:08:01.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836489 2024-04-25T08:08:01.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836489 2024-04-25T08:08:01.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489' 2024-04-25T08:08:01.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:08:01.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:08:01.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672966 2024-04-25T08:08:01.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672966 2024-04-25T08:08:01.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489 1-42949672966' 2024-04-25T08:08:01.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:08:01.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:08:01.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836489 2024-04-25T08:08:01.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:08:01.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836489 2024-04-25T08:08:01.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:08:01.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836489 2024-04-25T08:08:01.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-04-25T08:08:01.819 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836489 2024-04-25T08:08:01.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:08:02.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836486 -lt 21474836489 2024-04-25T08:08:02.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:08:03.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:08:03.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:08:03.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836489 -lt 21474836489 2024-04-25T08:08:03.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:08:03.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672966 2024-04-25T08:08:03.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:08:03.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:08:03.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672966 2024-04-25T08:08:03.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:08:03.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672966 2024-04-25T08:08:03.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2024-04-25T08:08:03.644 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672966 2024-04-25T08:08:03.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:08:04.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672966 -lt 42949672966 2024-04-25T08:08:04.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:08:04.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:04.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:04.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T08:08:04.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:04.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:04.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:04.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:04.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:04.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:04.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:05.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:08:05.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:05.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:05.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:05.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:08:05.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:08:05.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:08:05.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:348: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:08:06.050 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:08:06.050 INFO:tasks.workunit.client.0.smithi134.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:56.529636+0000 0'0 26:18 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:07:53.223622+0000 0'0 2024-04-25T08:07:53.223622+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:30:57.897459+0000 0 0 2024-04-25T08:08:06.050 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:54.019678+0000 0'0 26:28 [0,1] 0 [0,1] 0 0'0 2024-04-25T08:07:50.353552+0000 0'0 2024-04-25T08:07:50.353552+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:02:50.128492+0000 0 0 2024-04-25T08:08:06.050 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:07:50.197952+0000 0'0 26:37 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:07:46.938171+0000 0'0 2024-04-25T08:07:46.938171+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:47:04.106437+0000 0 0 2024-04-25T08:08:06.050 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:08:06.050 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:08:06.051 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:352: TEST_recovery_pool_priority: local PG1 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:353: TEST_recovery_pool_priority: local POOLNUM1 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:354: TEST_recovery_pool_priority: local pool1 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:355: TEST_recovery_pool_priority: local chk_osd1_1 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:356: TEST_recovery_pool_priority: local chk_osd1_2 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:358: TEST_recovery_pool_priority: local PG2 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:359: TEST_recovery_pool_priority: local POOLNUM2 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:360: TEST_recovery_pool_priority: local pool2 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:361: TEST_recovery_pool_priority: local chk_osd2_1 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:362: TEST_recovery_pool_priority: local chk_osd2_2 2024-04-25T08:08:06.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: seq 1 3 2024-04-25T08:08:06.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:06.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 1.0 --format=json 2024-04-25T08:08:06.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-04-25T08:08:06.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:08:06.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=1 2024-04-25T08:08:06.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:08:06.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=0 2024-04-25T08:08:06.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z '' ']' 2024-04-25T08:08:06.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:371: TEST_recovery_pool_priority: PG1=1.0 2024-04-25T08:08:06.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:372: TEST_recovery_pool_priority: POOLNUM1=1 2024-04-25T08:08:06.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:373: TEST_recovery_pool_priority: pool1=test1 2024-04-25T08:08:06.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:374: TEST_recovery_pool_priority: chk_osd1_1=1 2024-04-25T08:08:06.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:375: TEST_recovery_pool_priority: chk_osd1_2=0 2024-04-25T08:08:06.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:364: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:06.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: ceph pg map 2.0 --format=json 2024-04-25T08:08:06.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:366: TEST_recovery_pool_priority: jq '.acting[]' 2024-04-25T08:08:06.977 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:08:06.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:367: TEST_recovery_pool_priority: local test_osd1=0 2024-04-25T08:08:06.978 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:08:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:368: TEST_recovery_pool_priority: local test_osd2=1 2024-04-25T08:08:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:369: TEST_recovery_pool_priority: '[' -z 1.0 ']' 2024-04-25T08:08:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:376: TEST_recovery_pool_priority: '[' 1 '!=' 0 ']' 2024-04-25T08:08:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:378: TEST_recovery_pool_priority: PG2=2.0 2024-04-25T08:08:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:379: TEST_recovery_pool_priority: POOLNUM2=2 2024-04-25T08:08:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:380: TEST_recovery_pool_priority: pool2=test2 2024-04-25T08:08:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:381: TEST_recovery_pool_priority: chk_osd2_1=0 2024-04-25T08:08:06.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:382: TEST_recovery_pool_priority: chk_osd2_2=1 2024-04-25T08:08:06.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:383: TEST_recovery_pool_priority: break 2024-04-25T08:08:06.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:386: TEST_recovery_pool_priority: rm -f td/osd-recovery-prio/acting 2024-04-25T08:08:06.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:388: TEST_recovery_pool_priority: '[' test2 = '' ']' 2024-04-25T08:08:06.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: seq 1 3 2024-04-25T08:08:06.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:06.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2024-04-25T08:08:06.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:06.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2024-04-25T08:08:06.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:394: TEST_recovery_pool_priority: for p in $(seq 1 $pools) 2024-04-25T08:08:06.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:396: TEST_recovery_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2024-04-25T08:08:06.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:398: TEST_recovery_pool_priority: delete_pool test3 2024-04-25T08:08:06.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-04-25T08:08:06.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-04-25T08:08:07.607 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test3' does not exist 2024-04-25T08:08:07.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:402: TEST_recovery_pool_priority: pool1_extra_prio=1 2024-04-25T08:08:07.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:403: TEST_recovery_pool_priority: pool2_extra_prio=2 2024-04-25T08:08:07.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: expr 190 + 1 2024-04-25T08:08:07.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:404: TEST_recovery_pool_priority: pool1_prio=191 2024-04-25T08:08:07.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: expr 190 + 2 2024-04-25T08:08:07.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:405: TEST_recovery_pool_priority: pool2_prio=192 2024-04-25T08:08:07.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:407: TEST_recovery_pool_priority: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-04-25T08:08:08.988 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 1 2024-04-25T08:08:09.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:408: TEST_recovery_pool_priority: ceph osd pool set test1 recovery_priority 1 2024-04-25T08:08:10.075 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 recovery_priority to 1 2024-04-25T08:08:10.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:409: TEST_recovery_pool_priority: ceph osd pool set test2 size 1 --yes-i-really-mean-it 2024-04-25T08:08:11.171 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 2 size to 1 2024-04-25T08:08:11.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:410: TEST_recovery_pool_priority: ceph osd pool set test2 recovery_priority 2 2024-04-25T08:08:12.618 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 2 recovery_priority to 2 2024-04-25T08:08:12.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:411: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T08:08:12.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:08:12.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:08:12.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:08:12.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:08:12.644 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:08:12.644 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:08:12.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:08:12.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:08:12.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:08:12.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:08:12.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:08:12.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:08:12.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:08:12.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:08:12.758 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:08:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:08:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T08:08:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:08:13.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:08:13.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:08:13.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836493 2024-04-25T08:08:13.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836493 2024-04-25T08:08:13.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493' 2024-04-25T08:08:13.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:08:13.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:08:13.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672970 2024-04-25T08:08:13.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672970 2024-04-25T08:08:13.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836493 1-42949672970' 2024-04-25T08:08:13.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:08:13.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836493 2024-04-25T08:08:13.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:08:13.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:08:13.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836493 2024-04-25T08:08:13.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:08:13.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836493 2024-04-25T08:08:13.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2024-04-25T08:08:13.496 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836493 2024-04-25T08:08:13.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:08:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836491 -lt 21474836493 2024-04-25T08:08:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:08:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:08:14.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:08:15.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836493 -lt 21474836493 2024-04-25T08:08:15.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:08:15.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:08:15.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672970 2024-04-25T08:08:15.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:08:15.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672970 2024-04-25T08:08:15.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:08:15.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672970 2024-04-25T08:08:15.323 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672970 2024-04-25T08:08:15.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2024-04-25T08:08:15.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:08:15.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672970 -lt 42949672970 2024-04-25T08:08:15.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:08:15.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:15.770 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:16.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T08:08:16.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:08:16.293 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:08:16.293 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:08:16.293 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:08:16.293 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:08:16.293 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:08:16.293 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:08:16.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:08:16.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:08:16.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:08:16.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:08:17.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T08:08:17.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:08:17.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:08:17.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:413: TEST_recovery_pool_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-04-25T08:08:17.320 INFO:tasks.workunit.client.0.smithi134.stderr:10+0 records in 2024-04-25T08:08:17.320 INFO:tasks.workunit.client.0.smithi134.stderr:10+0 records out 2024-04-25T08:08:17.321 INFO:tasks.workunit.client.0.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0386503 s, 271 MB/s 2024-04-25T08:08:17.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:414: TEST_recovery_pool_priority: p=1 2024-04-25T08:08:17.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-04-25T08:08:17.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-04-25T08:08:17.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:17.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-04-25T08:08:17.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:17.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-04-25T08:08:18.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:18.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-04-25T08:08:19.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:19.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-04-25T08:08:19.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:19.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-04-25T08:08:20.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:20.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-04-25T08:08:20.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:20.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-04-25T08:08:21.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:21.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-04-25T08:08:22.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:22.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-04-25T08:08:22.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:22.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-04-25T08:08:23.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:23.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-04-25T08:08:23.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:23.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-04-25T08:08:24.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:24.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-04-25T08:08:25.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:25.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-04-25T08:08:25.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:25.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-04-25T08:08:26.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:26.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-04-25T08:08:27.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:27.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-04-25T08:08:27.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:27.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-04-25T08:08:28.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:28.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-04-25T08:08:28.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:28.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-04-25T08:08:29.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:29.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-04-25T08:08:29.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:29.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-04-25T08:08:30.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:30.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-04-25T08:08:31.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:31.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-04-25T08:08:31.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:31.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-04-25T08:08:32.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:32.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-04-25T08:08:32.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:32.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-04-25T08:08:33.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:33.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-04-25T08:08:33.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:33.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-04-25T08:08:34.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:34.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-04-25T08:08:35.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:35.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-04-25T08:08:35.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:35.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-04-25T08:08:36.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:36.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-04-25T08:08:36.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:36.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-04-25T08:08:37.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:37.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-04-25T08:08:38.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:38.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-04-25T08:08:38.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:38.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-04-25T08:08:39.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:39.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-04-25T08:08:39.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:39.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-04-25T08:08:40.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:40.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-04-25T08:08:41.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:41.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-04-25T08:08:41.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:41.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-04-25T08:08:42.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:42.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-04-25T08:08:43.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:43.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-04-25T08:08:43.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:43.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-04-25T08:08:44.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:44.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-04-25T08:08:44.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:44.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-04-25T08:08:45.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:45.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-04-25T08:08:45.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:45.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-04-25T08:08:46.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:46.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-04-25T08:08:47.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:47.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-04-25T08:08:47.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:47.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-04-25T08:08:48.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:48.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-04-25T08:08:49.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:49.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-04-25T08:08:49.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:49.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-04-25T08:08:50.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:50.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-04-25T08:08:50.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:50.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-04-25T08:08:51.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:51.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-04-25T08:08:52.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:52.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-04-25T08:08:52.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:52.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-04-25T08:08:53.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:53.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-04-25T08:08:54.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:54.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-04-25T08:08:54.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:54.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-04-25T08:08:55.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:55.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-04-25T08:08:55.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:55.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-04-25T08:08:56.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:56.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-04-25T08:08:57.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:57.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-04-25T08:08:57.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:57.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-04-25T08:08:58.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:58.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-04-25T08:08:58.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:58.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-04-25T08:08:59.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:08:59.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-04-25T08:09:00.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:00.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-04-25T08:09:00.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:00.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-04-25T08:09:01.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:01.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-04-25T08:09:01.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:01.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-04-25T08:09:02.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:02.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-04-25T08:09:03.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:03.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-04-25T08:09:03.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:03.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-04-25T08:09:04.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:04.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-04-25T08:09:04.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:04.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-04-25T08:09:05.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:05.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-04-25T08:09:06.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:06.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-04-25T08:09:06.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:06.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-04-25T08:09:07.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:07.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-04-25T08:09:07.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:07.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-04-25T08:09:08.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:08.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-04-25T08:09:09.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:09.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-04-25T08:09:09.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:09.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-04-25T08:09:10.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:10.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-04-25T08:09:11.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:11.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-04-25T08:09:11.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:11.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-04-25T08:09:12.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:12.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-04-25T08:09:12.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:12.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-04-25T08:09:13.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:13.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-04-25T08:09:14.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:14.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-04-25T08:09:14.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:14.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-04-25T08:09:15.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:15.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-04-25T08:09:15.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:15.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-04-25T08:09:16.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:16.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-04-25T08:09:17.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:17.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-04-25T08:09:17.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:17.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-04-25T08:09:18.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:18.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-04-25T08:09:18.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:18.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-04-25T08:09:19.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:19.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-04-25T08:09:20.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:20.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-04-25T08:09:20.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:20.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-04-25T08:09:21.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:21.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-04-25T08:09:21.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:21.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-04-25T08:09:22.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:22.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-04-25T08:09:23.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:23.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-04-25T08:09:23.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:23.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-04-25T08:09:24.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:24.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-04-25T08:09:25.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:25.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-04-25T08:09:25.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:25.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-04-25T08:09:26.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:26.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-04-25T08:09:26.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:26.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-04-25T08:09:27.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:27.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-04-25T08:09:27.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:27.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-04-25T08:09:28.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:28.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-04-25T08:09:29.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:29.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-04-25T08:09:29.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:29.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-04-25T08:09:30.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:30.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-04-25T08:09:30.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:30.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-04-25T08:09:31.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:31.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-04-25T08:09:32.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:32.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-04-25T08:09:32.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:32.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-04-25T08:09:33.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:33.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-04-25T08:09:33.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:33.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-04-25T08:09:34.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:34.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-04-25T08:09:35.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:35.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-04-25T08:09:35.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:35.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-04-25T08:09:36.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:36.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-04-25T08:09:36.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:36.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-04-25T08:09:37.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:37.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-04-25T08:09:37.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:37.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-04-25T08:09:38.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:38.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-04-25T08:09:39.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:39.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-04-25T08:09:39.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:39.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-04-25T08:09:40.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:40.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-04-25T08:09:41.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:41.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-04-25T08:09:41.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:41.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-04-25T08:09:42.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:42.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-04-25T08:09:42.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:42.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-04-25T08:09:43.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:43.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-04-25T08:09:43.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:43.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-04-25T08:09:44.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:44.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-04-25T08:09:45.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:45.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-04-25T08:09:45.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:45.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-04-25T08:09:46.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:46.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-04-25T08:09:47.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:47.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-04-25T08:09:47.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:47.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-04-25T08:09:48.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:48.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-04-25T08:09:48.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:48.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-04-25T08:09:49.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:49.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-04-25T08:09:49.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:49.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-04-25T08:09:50.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:50.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-04-25T08:09:51.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:51.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-04-25T08:09:51.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:51.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-04-25T08:09:52.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:52.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-04-25T08:09:53.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:53.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-04-25T08:09:53.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:53.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-04-25T08:09:54.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:54.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-04-25T08:09:55.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:55.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-04-25T08:09:55.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:55.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-04-25T08:09:56.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:56.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-04-25T08:09:56.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:56.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-04-25T08:09:57.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:57.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-04-25T08:09:57.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:57.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-04-25T08:09:58.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:58.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-04-25T08:09:59.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:59.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-04-25T08:09:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:09:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-04-25T08:10:00.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:00.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-04-25T08:10:01.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:01.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-04-25T08:10:01.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:01.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-04-25T08:10:02.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:02.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-04-25T08:10:02.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:02.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-04-25T08:10:03.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:03.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-04-25T08:10:03.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:03.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-04-25T08:10:04.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:04.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-04-25T08:10:05.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:05.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-04-25T08:10:05.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:05.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-04-25T08:10:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-04-25T08:10:07.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:07.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-04-25T08:10:07.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:07.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-04-25T08:10:08.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:08.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-04-25T08:10:08.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:08.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-04-25T08:10:09.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:09.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-04-25T08:10:09.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:09.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-04-25T08:10:10.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:10.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-04-25T08:10:11.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:11.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-04-25T08:10:11.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:11.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-04-25T08:10:12.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:12.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-04-25T08:10:12.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:12.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-04-25T08:10:13.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:13.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-04-25T08:10:14.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:14.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-04-25T08:10:14.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:14.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-04-25T08:10:15.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:15.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-04-25T08:10:15.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:15.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-04-25T08:10:16.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:16.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-04-25T08:10:17.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:17.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-04-25T08:10:17.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 1 + 1 2024-04-25T08:10:17.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=2 2024-04-25T08:10:17.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:415: TEST_recovery_pool_priority: for pname in $pool1 $pool2 2024-04-25T08:10:17.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: seq 1 200 2024-04-25T08:10:17.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:17.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj1-p2 td/osd-recovery-prio/data 2024-04-25T08:10:18.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:18.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj2-p2 td/osd-recovery-prio/data 2024-04-25T08:10:19.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:19.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj3-p2 td/osd-recovery-prio/data 2024-04-25T08:10:19.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:19.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj4-p2 td/osd-recovery-prio/data 2024-04-25T08:10:20.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:20.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj5-p2 td/osd-recovery-prio/data 2024-04-25T08:10:20.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:20.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj6-p2 td/osd-recovery-prio/data 2024-04-25T08:10:21.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:21.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj7-p2 td/osd-recovery-prio/data 2024-04-25T08:10:22.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:22.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj8-p2 td/osd-recovery-prio/data 2024-04-25T08:10:22.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:22.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj9-p2 td/osd-recovery-prio/data 2024-04-25T08:10:23.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:23.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj10-p2 td/osd-recovery-prio/data 2024-04-25T08:10:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj11-p2 td/osd-recovery-prio/data 2024-04-25T08:10:24.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:24.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj12-p2 td/osd-recovery-prio/data 2024-04-25T08:10:25.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:25.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj13-p2 td/osd-recovery-prio/data 2024-04-25T08:10:25.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:25.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj14-p2 td/osd-recovery-prio/data 2024-04-25T08:10:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:26.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj15-p2 td/osd-recovery-prio/data 2024-04-25T08:10:27.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:27.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj16-p2 td/osd-recovery-prio/data 2024-04-25T08:10:27.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:27.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj17-p2 td/osd-recovery-prio/data 2024-04-25T08:10:28.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:28.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj18-p2 td/osd-recovery-prio/data 2024-04-25T08:10:28.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:28.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj19-p2 td/osd-recovery-prio/data 2024-04-25T08:10:29.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:29.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj20-p2 td/osd-recovery-prio/data 2024-04-25T08:10:29.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:29.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj21-p2 td/osd-recovery-prio/data 2024-04-25T08:10:30.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:30.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj22-p2 td/osd-recovery-prio/data 2024-04-25T08:10:31.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:31.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj23-p2 td/osd-recovery-prio/data 2024-04-25T08:10:31.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:31.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj24-p2 td/osd-recovery-prio/data 2024-04-25T08:10:32.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:32.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj25-p2 td/osd-recovery-prio/data 2024-04-25T08:10:32.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:32.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj26-p2 td/osd-recovery-prio/data 2024-04-25T08:10:33.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:33.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj27-p2 td/osd-recovery-prio/data 2024-04-25T08:10:34.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:34.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj28-p2 td/osd-recovery-prio/data 2024-04-25T08:10:34.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:34.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj29-p2 td/osd-recovery-prio/data 2024-04-25T08:10:35.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:35.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj30-p2 td/osd-recovery-prio/data 2024-04-25T08:10:35.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:35.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj31-p2 td/osd-recovery-prio/data 2024-04-25T08:10:36.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:36.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj32-p2 td/osd-recovery-prio/data 2024-04-25T08:10:37.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:37.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj33-p2 td/osd-recovery-prio/data 2024-04-25T08:10:37.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:37.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj34-p2 td/osd-recovery-prio/data 2024-04-25T08:10:38.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:38.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj35-p2 td/osd-recovery-prio/data 2024-04-25T08:10:38.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:38.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj36-p2 td/osd-recovery-prio/data 2024-04-25T08:10:39.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:39.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj37-p2 td/osd-recovery-prio/data 2024-04-25T08:10:39.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:39.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj38-p2 td/osd-recovery-prio/data 2024-04-25T08:10:40.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:40.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj39-p2 td/osd-recovery-prio/data 2024-04-25T08:10:41.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:41.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj40-p2 td/osd-recovery-prio/data 2024-04-25T08:10:41.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:41.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj41-p2 td/osd-recovery-prio/data 2024-04-25T08:10:42.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:42.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj42-p2 td/osd-recovery-prio/data 2024-04-25T08:10:43.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:43.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj43-p2 td/osd-recovery-prio/data 2024-04-25T08:10:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:43.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj44-p2 td/osd-recovery-prio/data 2024-04-25T08:10:44.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:44.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj45-p2 td/osd-recovery-prio/data 2024-04-25T08:10:44.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:44.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj46-p2 td/osd-recovery-prio/data 2024-04-25T08:10:45.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:45.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj47-p2 td/osd-recovery-prio/data 2024-04-25T08:10:46.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:46.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj48-p2 td/osd-recovery-prio/data 2024-04-25T08:10:46.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:46.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj49-p2 td/osd-recovery-prio/data 2024-04-25T08:10:47.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:47.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj50-p2 td/osd-recovery-prio/data 2024-04-25T08:10:47.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:47.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj51-p2 td/osd-recovery-prio/data 2024-04-25T08:10:48.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:48.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj52-p2 td/osd-recovery-prio/data 2024-04-25T08:10:49.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:49.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj53-p2 td/osd-recovery-prio/data 2024-04-25T08:10:49.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:49.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj54-p2 td/osd-recovery-prio/data 2024-04-25T08:10:50.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:50.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj55-p2 td/osd-recovery-prio/data 2024-04-25T08:10:50.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:50.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj56-p2 td/osd-recovery-prio/data 2024-04-25T08:10:51.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:51.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj57-p2 td/osd-recovery-prio/data 2024-04-25T08:10:51.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:51.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj58-p2 td/osd-recovery-prio/data 2024-04-25T08:10:52.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:52.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj59-p2 td/osd-recovery-prio/data 2024-04-25T08:10:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj60-p2 td/osd-recovery-prio/data 2024-04-25T08:10:53.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:53.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj61-p2 td/osd-recovery-prio/data 2024-04-25T08:10:54.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:54.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj62-p2 td/osd-recovery-prio/data 2024-04-25T08:10:54.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:54.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj63-p2 td/osd-recovery-prio/data 2024-04-25T08:10:55.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:55.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj64-p2 td/osd-recovery-prio/data 2024-04-25T08:10:56.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:56.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj65-p2 td/osd-recovery-prio/data 2024-04-25T08:10:56.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:56.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj66-p2 td/osd-recovery-prio/data 2024-04-25T08:10:57.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:57.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj67-p2 td/osd-recovery-prio/data 2024-04-25T08:10:57.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:57.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj68-p2 td/osd-recovery-prio/data 2024-04-25T08:10:58.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:58.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj69-p2 td/osd-recovery-prio/data 2024-04-25T08:10:59.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:59.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj70-p2 td/osd-recovery-prio/data 2024-04-25T08:10:59.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:10:59.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj71-p2 td/osd-recovery-prio/data 2024-04-25T08:11:00.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:00.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj72-p2 td/osd-recovery-prio/data 2024-04-25T08:11:01.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:01.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj73-p2 td/osd-recovery-prio/data 2024-04-25T08:11:01.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:01.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj74-p2 td/osd-recovery-prio/data 2024-04-25T08:11:02.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:02.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj75-p2 td/osd-recovery-prio/data 2024-04-25T08:11:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj76-p2 td/osd-recovery-prio/data 2024-04-25T08:11:03.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:03.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj77-p2 td/osd-recovery-prio/data 2024-04-25T08:11:04.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:04.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj78-p2 td/osd-recovery-prio/data 2024-04-25T08:11:04.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:04.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj79-p2 td/osd-recovery-prio/data 2024-04-25T08:11:05.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:05.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj80-p2 td/osd-recovery-prio/data 2024-04-25T08:11:05.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:05.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj81-p2 td/osd-recovery-prio/data 2024-04-25T08:11:06.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:06.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj82-p2 td/osd-recovery-prio/data 2024-04-25T08:11:07.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:07.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj83-p2 td/osd-recovery-prio/data 2024-04-25T08:11:07.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:07.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj84-p2 td/osd-recovery-prio/data 2024-04-25T08:11:08.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:08.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj85-p2 td/osd-recovery-prio/data 2024-04-25T08:11:09.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:09.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj86-p2 td/osd-recovery-prio/data 2024-04-25T08:11:09.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:09.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj87-p2 td/osd-recovery-prio/data 2024-04-25T08:11:10.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:10.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj88-p2 td/osd-recovery-prio/data 2024-04-25T08:11:10.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:10.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj89-p2 td/osd-recovery-prio/data 2024-04-25T08:11:11.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:11.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj90-p2 td/osd-recovery-prio/data 2024-04-25T08:11:12.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:12.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj91-p2 td/osd-recovery-prio/data 2024-04-25T08:11:12.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:12.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj92-p2 td/osd-recovery-prio/data 2024-04-25T08:11:13.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:13.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj93-p2 td/osd-recovery-prio/data 2024-04-25T08:11:13.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:13.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj94-p2 td/osd-recovery-prio/data 2024-04-25T08:11:14.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:14.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj95-p2 td/osd-recovery-prio/data 2024-04-25T08:11:15.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:15.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj96-p2 td/osd-recovery-prio/data 2024-04-25T08:11:15.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:15.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj97-p2 td/osd-recovery-prio/data 2024-04-25T08:11:16.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:16.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj98-p2 td/osd-recovery-prio/data 2024-04-25T08:11:16.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:16.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj99-p2 td/osd-recovery-prio/data 2024-04-25T08:11:17.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:17.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj100-p2 td/osd-recovery-prio/data 2024-04-25T08:11:18.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:18.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj101-p2 td/osd-recovery-prio/data 2024-04-25T08:11:18.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:18.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj102-p2 td/osd-recovery-prio/data 2024-04-25T08:11:19.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:19.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj103-p2 td/osd-recovery-prio/data 2024-04-25T08:11:19.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:19.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj104-p2 td/osd-recovery-prio/data 2024-04-25T08:11:20.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:20.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj105-p2 td/osd-recovery-prio/data 2024-04-25T08:11:21.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:21.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj106-p2 td/osd-recovery-prio/data 2024-04-25T08:11:21.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:21.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj107-p2 td/osd-recovery-prio/data 2024-04-25T08:11:22.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:22.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj108-p2 td/osd-recovery-prio/data 2024-04-25T08:11:23.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:23.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj109-p2 td/osd-recovery-prio/data 2024-04-25T08:11:23.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:23.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj110-p2 td/osd-recovery-prio/data 2024-04-25T08:11:24.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:24.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj111-p2 td/osd-recovery-prio/data 2024-04-25T08:11:25.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:25.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj112-p2 td/osd-recovery-prio/data 2024-04-25T08:11:25.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:25.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj113-p2 td/osd-recovery-prio/data 2024-04-25T08:11:26.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:26.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj114-p2 td/osd-recovery-prio/data 2024-04-25T08:11:27.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:27.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj115-p2 td/osd-recovery-prio/data 2024-04-25T08:11:27.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:27.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj116-p2 td/osd-recovery-prio/data 2024-04-25T08:11:28.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:28.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj117-p2 td/osd-recovery-prio/data 2024-04-25T08:11:28.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:28.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj118-p2 td/osd-recovery-prio/data 2024-04-25T08:11:29.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:29.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj119-p2 td/osd-recovery-prio/data 2024-04-25T08:11:29.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:29.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj120-p2 td/osd-recovery-prio/data 2024-04-25T08:11:30.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:30.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj121-p2 td/osd-recovery-prio/data 2024-04-25T08:11:31.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:31.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj122-p2 td/osd-recovery-prio/data 2024-04-25T08:11:31.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:31.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj123-p2 td/osd-recovery-prio/data 2024-04-25T08:11:32.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:32.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj124-p2 td/osd-recovery-prio/data 2024-04-25T08:11:33.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:33.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj125-p2 td/osd-recovery-prio/data 2024-04-25T08:11:33.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:33.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj126-p2 td/osd-recovery-prio/data 2024-04-25T08:11:34.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:34.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj127-p2 td/osd-recovery-prio/data 2024-04-25T08:11:34.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:34.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj128-p2 td/osd-recovery-prio/data 2024-04-25T08:11:35.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:35.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj129-p2 td/osd-recovery-prio/data 2024-04-25T08:11:36.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:36.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj130-p2 td/osd-recovery-prio/data 2024-04-25T08:11:36.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:36.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj131-p2 td/osd-recovery-prio/data 2024-04-25T08:11:37.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:37.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj132-p2 td/osd-recovery-prio/data 2024-04-25T08:11:37.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:37.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj133-p2 td/osd-recovery-prio/data 2024-04-25T08:11:38.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:38.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj134-p2 td/osd-recovery-prio/data 2024-04-25T08:11:39.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:39.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj135-p2 td/osd-recovery-prio/data 2024-04-25T08:11:39.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:39.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj136-p2 td/osd-recovery-prio/data 2024-04-25T08:11:40.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:40.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj137-p2 td/osd-recovery-prio/data 2024-04-25T08:11:41.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:41.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj138-p2 td/osd-recovery-prio/data 2024-04-25T08:11:41.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:41.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj139-p2 td/osd-recovery-prio/data 2024-04-25T08:11:42.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:42.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj140-p2 td/osd-recovery-prio/data 2024-04-25T08:11:42.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:42.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj141-p2 td/osd-recovery-prio/data 2024-04-25T08:11:43.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:43.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj142-p2 td/osd-recovery-prio/data 2024-04-25T08:11:44.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:44.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj143-p2 td/osd-recovery-prio/data 2024-04-25T08:11:44.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:44.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj144-p2 td/osd-recovery-prio/data 2024-04-25T08:11:45.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:45.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj145-p2 td/osd-recovery-prio/data 2024-04-25T08:11:45.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:45.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj146-p2 td/osd-recovery-prio/data 2024-04-25T08:11:46.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:46.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj147-p2 td/osd-recovery-prio/data 2024-04-25T08:11:47.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:47.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj148-p2 td/osd-recovery-prio/data 2024-04-25T08:11:47.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:47.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj149-p2 td/osd-recovery-prio/data 2024-04-25T08:11:48.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:48.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj150-p2 td/osd-recovery-prio/data 2024-04-25T08:11:48.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:48.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj151-p2 td/osd-recovery-prio/data 2024-04-25T08:11:49.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:49.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj152-p2 td/osd-recovery-prio/data 2024-04-25T08:11:50.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:50.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj153-p2 td/osd-recovery-prio/data 2024-04-25T08:11:50.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:50.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj154-p2 td/osd-recovery-prio/data 2024-04-25T08:11:51.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:51.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj155-p2 td/osd-recovery-prio/data 2024-04-25T08:11:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj156-p2 td/osd-recovery-prio/data 2024-04-25T08:11:52.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:52.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj157-p2 td/osd-recovery-prio/data 2024-04-25T08:11:53.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:53.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj158-p2 td/osd-recovery-prio/data 2024-04-25T08:11:53.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj159-p2 td/osd-recovery-prio/data 2024-04-25T08:11:54.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:54.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj160-p2 td/osd-recovery-prio/data 2024-04-25T08:11:54.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:54.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj161-p2 td/osd-recovery-prio/data 2024-04-25T08:11:55.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:55.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj162-p2 td/osd-recovery-prio/data 2024-04-25T08:11:56.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:56.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj163-p2 td/osd-recovery-prio/data 2024-04-25T08:11:56.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:56.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj164-p2 td/osd-recovery-prio/data 2024-04-25T08:11:57.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:57.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj165-p2 td/osd-recovery-prio/data 2024-04-25T08:11:58.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:58.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj166-p2 td/osd-recovery-prio/data 2024-04-25T08:11:58.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:58.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj167-p2 td/osd-recovery-prio/data 2024-04-25T08:11:59.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:59.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj168-p2 td/osd-recovery-prio/data 2024-04-25T08:11:59.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:11:59.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj169-p2 td/osd-recovery-prio/data 2024-04-25T08:12:00.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:00.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj170-p2 td/osd-recovery-prio/data 2024-04-25T08:12:01.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:01.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj171-p2 td/osd-recovery-prio/data 2024-04-25T08:12:01.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:01.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj172-p2 td/osd-recovery-prio/data 2024-04-25T08:12:02.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:02.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj173-p2 td/osd-recovery-prio/data 2024-04-25T08:12:03.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:03.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj174-p2 td/osd-recovery-prio/data 2024-04-25T08:12:03.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:03.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj175-p2 td/osd-recovery-prio/data 2024-04-25T08:12:04.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:04.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj176-p2 td/osd-recovery-prio/data 2024-04-25T08:12:04.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:04.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj177-p2 td/osd-recovery-prio/data 2024-04-25T08:12:05.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:05.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj178-p2 td/osd-recovery-prio/data 2024-04-25T08:12:05.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:05.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj179-p2 td/osd-recovery-prio/data 2024-04-25T08:12:06.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:06.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj180-p2 td/osd-recovery-prio/data 2024-04-25T08:12:07.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:07.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj181-p2 td/osd-recovery-prio/data 2024-04-25T08:12:07.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:07.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj182-p2 td/osd-recovery-prio/data 2024-04-25T08:12:08.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:08.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj183-p2 td/osd-recovery-prio/data 2024-04-25T08:12:09.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:09.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj184-p2 td/osd-recovery-prio/data 2024-04-25T08:12:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj185-p2 td/osd-recovery-prio/data 2024-04-25T08:12:10.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:10.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj186-p2 td/osd-recovery-prio/data 2024-04-25T08:12:10.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:10.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj187-p2 td/osd-recovery-prio/data 2024-04-25T08:12:11.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:11.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj188-p2 td/osd-recovery-prio/data 2024-04-25T08:12:12.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:12.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj189-p2 td/osd-recovery-prio/data 2024-04-25T08:12:12.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:12.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj190-p2 td/osd-recovery-prio/data 2024-04-25T08:12:13.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:13.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj191-p2 td/osd-recovery-prio/data 2024-04-25T08:12:13.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:13.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj192-p2 td/osd-recovery-prio/data 2024-04-25T08:12:14.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:14.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj193-p2 td/osd-recovery-prio/data 2024-04-25T08:12:15.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:15.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj194-p2 td/osd-recovery-prio/data 2024-04-25T08:12:15.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:15.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj195-p2 td/osd-recovery-prio/data 2024-04-25T08:12:16.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:16.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj196-p2 td/osd-recovery-prio/data 2024-04-25T08:12:16.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:16.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj197-p2 td/osd-recovery-prio/data 2024-04-25T08:12:17.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:17.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj198-p2 td/osd-recovery-prio/data 2024-04-25T08:12:18.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:18.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj199-p2 td/osd-recovery-prio/data 2024-04-25T08:12:18.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:417: TEST_recovery_pool_priority: for i in $(seq 1 $objects) 2024-04-25T08:12:18.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:419: TEST_recovery_pool_priority: rados -p test2 put obj200-p2 td/osd-recovery-prio/data 2024-04-25T08:12:19.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: expr 2 + 1 2024-04-25T08:12:19.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:421: TEST_recovery_pool_priority: p=3 2024-04-25T08:12:19.361 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: get_not_primary test1 obj1-p1 2024-04-25T08:12:19.361 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-04-25T08:12:19.361 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-04-25T08:12:19.362 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-04-25T08:12:19.362 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-04-25T08:12:19.362 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-04-25T08:12:19.363 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:12:19.363 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:12:19.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:12:19.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:12:19.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:12:20.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:424: TEST_recovery_pool_priority: local otherosd=null 2024-04-25T08:12:20.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:426: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:12:20.693 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:12:20.693 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 196 0 0 0 0 2048917504 0 0 586 0 586 active+clean 2024-04-25T08:08:12.019464+0000 36'586 36:639 [0] 0 [0] 0 0'0 2024-04-25T08:07:50.353552+0000 0'0 2024-04-25T08:07:50.353552+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:52:45.792592+0000 0 0 2024-04-25T08:12:20.693 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:08:09.291504+0000 36'600 36:665 [1] 1 [1] 1 0'0 2024-04-25T08:07:46.938171+0000 0'0 2024-04-25T08:07:46.938171+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:33:57.184453+0000 0 0 2024-04-25T08:12:20.693 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:12:20.693 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:12:20.694 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:12:20.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:427: TEST_recovery_pool_priority: ERRORS=0 2024-04-25T08:12:20.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:429: TEST_recovery_pool_priority: ceph osd pool set test1 size 2 2024-04-25T08:12:21.474 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 2 2024-04-25T08:12:21.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:430: TEST_recovery_pool_priority: ceph osd pool set test2 size 2 2024-04-25T08:12:22.453 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 2 size to 2 2024-04-25T08:12:22.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:433: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:12:23.029 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:12:23.029 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:08:12.019464+0000 36'600 37:655 [0] 0 [0] 0 0'0 2024-04-25T08:07:50.353552+0000 0'0 2024-04-25T08:07:50.353552+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:52:45.792592+0000 0 0 2024-04-25T08:12:23.029 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:08:09.291504+0000 36'600 36:665 [1] 1 [1] 1 0'0 2024-04-25T08:07:46.938171+0000 0'0 2024-04-25T08:07:46.938171+0000 0 0 periodic scrub scheduled @ 2024-04-26T08:33:57.184453+0000 0 0 2024-04-25T08:12:23.029 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:12:23.029 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:12:23.029 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:12:23.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:436: TEST_recovery_pool_priority: set -o pipefail 2024-04-25T08:12:23.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:437: TEST_recovery_pool_priority: count=0 2024-04-25T08:12:23.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T08:12:23.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T08:12:23.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T08:12:23.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T08:12:23.722 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:12:23.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2024-04-25T08:12:23.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-04-25T08:12:25.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 0 -eq 10 2024-04-25T08:12:25.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 0 + 1 2024-04-25T08:12:25.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=1 2024-04-25T08:12:25.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T08:12:25.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T08:12:25.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T08:12:25.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T08:12:26.199 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:12:26.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 0 == 2 2024-04-25T08:12:26.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:445: TEST_recovery_pool_priority: sleep 2 2024-04-25T08:12:28.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:446: TEST_recovery_pool_priority: test 1 -eq 10 2024-04-25T08:12:28.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: expr 1 + 1 2024-04-25T08:12:28.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:451: TEST_recovery_pool_priority: count=2 2024-04-25T08:12:28.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:438: TEST_recovery_pool_priority: true 2024-04-25T08:12:28.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: ceph --format json pg dump pgs 2024-04-25T08:12:28.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: jq '.pg_stats | .[] | .state | contains("recovering")' 2024-04-25T08:12:28.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:442: TEST_recovery_pool_priority: grep -c true 2024-04-25T08:12:28.635 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:12:28.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:441: TEST_recovery_pool_priority: test 2 == 2 2024-04-25T08:12:28.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:443: TEST_recovery_pool_priority: break 2024-04-25T08:12:28.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:453: TEST_recovery_pool_priority: set +o pipefail 2024-04-25T08:12:28.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:454: TEST_recovery_pool_priority: ceph pg dump pgs 2024-04-25T08:12:29.311 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:12:29.311 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 200 0 199 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-04-25T08:12:25.141473+0000 36'600 43:669 [0,1] 0 [0] 0 0'0 2024-04-25T08:07:50.353552+0000 0'0 2024-04-25T08:07:50.353552+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:12:29.311 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 200 0 198 0 0 2097152000 0 0 600 0 600 active+recovering+undersized+degraded+remapped 2024-04-25T08:12:23.939555+0000 36'600 43:681 [1,0] 1 [1] 1 0'0 2024-04-25T08:07:46.938171+0000 0'0 2024-04-25T08:07:46.938171+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:12:29.311 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:12:29.311 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:12:29.311 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:12:29.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: get_asok_path osd.1 2024-04-25T08:12:29.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:12:29.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:12:29.329 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:12:29.329 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:12:29.329 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:12:29.330 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:12:29.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: CEPH_ARGS= 2024-04-25T08:12:29.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:456: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:12:29.439 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 2024-04-25T08:12:29.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:457: TEST_recovery_pool_priority: echo osd.1 2024-04-25T08:12:29.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:458: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:12:29.440 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:12:29.440 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:12:29.440 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:12:29.440 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:12:29.440 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:12:29.440 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:12:29.440 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 191, 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "2.0", 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 192, 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:12:29.441 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:12:29.442 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:12:29.442 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:12:29.442 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:12:29.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: get_asok_path osd.0 2024-04-25T08:12:29.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T08:12:29.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T08:12:29.443 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:12:29.443 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:12:29.443 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:12:29.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.0.asok 2024-04-25T08:12:29.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: CEPH_ARGS= 2024-04-25T08:12:29.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:459: TEST_recovery_pool_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.0.asok dump_recovery_reservations 2024-04-25T08:12:29.571 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 2024-04-25T08:12:29.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:460: TEST_recovery_pool_priority: echo osd.0 2024-04-25T08:12:29.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:461: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:12:29.572 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "2.0", 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 192, 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:12:29.573 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 191, 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:12:29.574 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:12:29.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:12:29.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:12:29.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-04-25T08:12:29.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:464: TEST_recovery_pool_priority: ITEM=1.0 2024-04-25T08:12:29.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:465: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-04-25T08:12:29.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:12:29.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:12:29.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:470: TEST_recovery_pool_priority: PRIO=191 2024-04-25T08:12:29.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:471: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-04-25T08:12:29.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:12:29.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-04-25T08:12:29.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: eval 'ITEM="1.0"' 2024-04-25T08:12:29.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:479: TEST_recovery_pool_priority: ITEM=1.0 2024-04-25T08:12:29.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:480: TEST_recovery_pool_priority: '[' 1.0 '!=' 1.0 ']' 2024-04-25T08:12:29.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-04-25T08:12:29.683 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:12:29.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:485: TEST_recovery_pool_priority: PRIO=191 2024-04-25T08:12:29.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:486: TEST_recovery_pool_priority: '[' 191 '!=' 191 ']' 2024-04-25T08:12:29.718 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:12:29.718 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:12:29.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-04-25T08:12:29.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:494: TEST_recovery_pool_priority: ITEM=2.0 2024-04-25T08:12:29.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:495: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-04-25T08:12:29.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.0.out 2024-04-25T08:12:29.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:12:29.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:500: TEST_recovery_pool_priority: PRIO=192 2024-04-25T08:12:29.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:501: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-04-25T08:12:29.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:12:29.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].item' 2024-04-25T08:12:29.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: eval 'ITEM="2.0"' 2024-04-25T08:12:29.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:509: TEST_recovery_pool_priority: ITEM=2.0 2024-04-25T08:12:29.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:510: TEST_recovery_pool_priority: '[' 2.0 '!=' 2.0 ']' 2024-04-25T08:12:29.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: cat td/osd-recovery-prio/dump.1.out 2024-04-25T08:12:29.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2024-04-25T08:12:29.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:515: TEST_recovery_pool_priority: PRIO=192 2024-04-25T08:12:29.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:516: TEST_recovery_pool_priority: '[' 192 '!=' 192 ']' 2024-04-25T08:12:29.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:523: TEST_recovery_pool_priority: wait_for_clean 2024-04-25T08:12:29.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:12:29.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:12:29.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:12:29.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:12:29.862 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:12:29.862 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:12:29.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:12:29.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:12:29.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:12:29.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:12:29.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:12:29.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:12:29.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:12:29.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:12:29.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:12:31.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:12:31.037 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T08:12:31.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:12:31.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:31.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:12:31.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836546 2024-04-25T08:12:31.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836546 2024-04-25T08:12:31.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836546' 2024-04-25T08:12:31.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:12:31.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:12:31.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673023 2024-04-25T08:12:31.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673023 2024-04-25T08:12:31.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836546 1-42949673023' 2024-04-25T08:12:31.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:31.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836546 2024-04-25T08:12:31.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:31.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:12:31.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836546 2024-04-25T08:12:31.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:31.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836546 2024-04-25T08:12:31.457 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836546 2024-04-25T08:12:31.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-04-25T08:12:31.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:12:31.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836544 -lt 21474836546 2024-04-25T08:12:31.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:12:32.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:12:32.865 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:12:33.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836547 -lt 21474836546 2024-04-25T08:12:33.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:12:33.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673023 2024-04-25T08:12:33.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:12:33.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:12:33.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673023 2024-04-25T08:12:33.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:12:33.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673023 2024-04-25T08:12:33.583 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673023 2024-04-25T08:12:33.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673023' 2024-04-25T08:12:33.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:12:34.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673024 -lt 42949673023 2024-04-25T08:12:34.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:12:34.270 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:34.270 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:35.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T08:12:35.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:35.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:35.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:35.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:35.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:35.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:35.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:35.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:35.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:35.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:35.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:36.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:12:36.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:12:36.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:12:36.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:12:36.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:12:36.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:12:37.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:12:37.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:37.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:37.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:37.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:37.012 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:37.012 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:37.012 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:37.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:37.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:37.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:37.770 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:38.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:12:38.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:12:38.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:12:38.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:12:38.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:12:38.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:12:38.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:12:38.333 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:12:38.333 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:12:39.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20964512 2024-04-25T08:12:39.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20964512 '!=' null 2024-04-25T08:12:39.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:12:39.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:12:39.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:12:39.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:12:39.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:39.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:39.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:39.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:39.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:39.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:39.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:40.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:40.308 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:40.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:40.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:40.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:12:40.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:12:40.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:12:40.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:12:40.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:12:40.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:12:40.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:12:40.933 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:12:40.933 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:12:41.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18343934 2024-04-25T08:12:41.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18343934 '!=' null 2024-04-25T08:12:41.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:12:41.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:12:41.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:12:41.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:12:41.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:41.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:41.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:41.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:41.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:41.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:41.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:42.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:42.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:42.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:42.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:43.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:12:43.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:12:43.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:12:43.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:12:43.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:12:43.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:12:43.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:12:43.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:12:43.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:12:44.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27952696 2024-04-25T08:12:44.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27952696 '!=' null 2024-04-25T08:12:44.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:12:44.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:12:44.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:12:44.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:12:44.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:44.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:44.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:44.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:44.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:44.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:44.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:45.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:45.081 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:45.081 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:45.081 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:45.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:12:45.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:12:45.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:12:45.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:12:45.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:12:45.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:12:45.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:12:45.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:12:45.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:12:47.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18343922 2024-04-25T08:12:47.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18343922 '!=' null 2024-04-25T08:12:47.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:12:47.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:12:47.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:12:47.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:12:47.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:47.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:47.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:47.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:47.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:47.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:47.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:48.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:48.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:48.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:48.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:49.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:12:49.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:12:49.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:12:49.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:12:49.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:12:49.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:12:49.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:12:49.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:12:49.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:12:50.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27952856 2024-04-25T08:12:50.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27952856 '!=' null 2024-04-25T08:12:50.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:12:50.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:12:50.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:12:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:12:50.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:50.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:50.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:50.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:50.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:50.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:50.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:50.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:50.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:50.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:50.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:12:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:12:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:12:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:12:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:12:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:12:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:12:51.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:12:51.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:12:53.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19217846 2024-04-25T08:12:53.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19217846 '!=' null 2024-04-25T08:12:53.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:12:53.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:12:53.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:12:53.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:12:53.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:53.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:53.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:53.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:53.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:53.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:53.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:53.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:53.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:53.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:53.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:54.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:12:54.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:12:54.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:12:54.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:12:54.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:12:54.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:12:54.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:12:54.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:12:54.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:12:56.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344839 2024-04-25T08:12:56.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344839 '!=' null 2024-04-25T08:12:56.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:12:56.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:12:56.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:12:56.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:12:56.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:56.118 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:56.118 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:56.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:56.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:56.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:56.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:56.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:56.604 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:56.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:12:56.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:58.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:12:58.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:12:58.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:12:58.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:12:58.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:12:58.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:12:58.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:12:58.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:12:58.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344999 2024-04-25T08:12:58.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344999 '!=' null 2024-04-25T08:12:58.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:12:58.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:12:58.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:12:59.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:12:59.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:12:59.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:12:59.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:12:59.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:12:59.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:12:59.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:12:59.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:12:59.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:12:59.654 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:12:59.654 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:12:59.654 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:00.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:00.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:00.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:00.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:00.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:00.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:00.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:00.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:00.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:02.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471556 2024-04-25T08:13:02.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471556 '!=' null 2024-04-25T08:13:02.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:02.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:02.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:02.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:02.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:02.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:02.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:02.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:02.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:02.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:02.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:02.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:02.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:02.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:02.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:03.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:03.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:03.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:03.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:03.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:03.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:03.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:03.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:03.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:05.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344891 2024-04-25T08:13:05.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344891 '!=' null 2024-04-25T08:13:05.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:05.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:05.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:05.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:05.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:05.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:05.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:05.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:05.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:05.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:05.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:06.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:06.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:06.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:06.090 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:07.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:07.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:07.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:07.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:07.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:07.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:07.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:07.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:07.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:07.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18345039 2024-04-25T08:13:07.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18345039 '!=' null 2024-04-25T08:13:07.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:07.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:07.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:07.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:07.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:07.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:07.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:07.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:07.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:07.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:07.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:08.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:08.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:08.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:08.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:09.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:09.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:09.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:09.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:09.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:09.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:09.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:09.667 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:09.667 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:10.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080993 2024-04-25T08:13:10.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080993 '!=' null 2024-04-25T08:13:10.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:10.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:10.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:10.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:10.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:10.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:10.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:10.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:10.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:10.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:10.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:11.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:11.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:11.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:11.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:12.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:12.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:12.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:12.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:12.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:12.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:12.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:12.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:12.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:13.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344807 2024-04-25T08:13:13.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344807 '!=' null 2024-04-25T08:13:13.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:13.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:13.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:13.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:13.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:13.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:13.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:13.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:13.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:13.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:13.115 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:13.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:13.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:13.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:13.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:14.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:14.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:14.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:14.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:14.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:14.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:14.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:14.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:14.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:15.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827514 2024-04-25T08:13:15.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827514 '!=' null 2024-04-25T08:13:15.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:15.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:15.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:15.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:15.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:15.690 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:15.690 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:15.690 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:15.690 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:15.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:15.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:16.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:16.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:16.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:16.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:17.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:17.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:17.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:17.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:17.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:17.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:17.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:17.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:17.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:18.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218647 2024-04-25T08:13:18.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218647 '!=' null 2024-04-25T08:13:18.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:18.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:18.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:18.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:18.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:18.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:18.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:18.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:18.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:18.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:18.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:19.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:19.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:19.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:19.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:20.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:20.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:20.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:20.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:20.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:20.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:20.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:20.012 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:20.012 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:21.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080928 2024-04-25T08:13:21.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080928 '!=' null 2024-04-25T08:13:21.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:21.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:21.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:21.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:21.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:21.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:21.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:21.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:21.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:21.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:21.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:21.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:21.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:21.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:21.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:22.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:22.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:22.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:22.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:22.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:22.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:22.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:22.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:22.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:24.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954319 2024-04-25T08:13:24.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954319 '!=' null 2024-04-25T08:13:24.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:24.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:24.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:24.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:24.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:24.196 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:24.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:24.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:24.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:24.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:24.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:24.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:24.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:24.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:24.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:25.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:25.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:25.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:25.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:25.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:25.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:25.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:25.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:25.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:25.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471224 2024-04-25T08:13:25.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471224 '!=' null 2024-04-25T08:13:25.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:25.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:25.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:26.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:26.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:26.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:26.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:26.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:26.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:26.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:26.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:26.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:26.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:26.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:26.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:27.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:27.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:27.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:27.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:27.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:27.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:27.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:27.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:27.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471056 2024-04-25T08:13:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471056 '!=' null 2024-04-25T08:13:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:28.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:28.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:28.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:28.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:28.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:28.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:28.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:28.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:29.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:29.330 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:29.330 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:29.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:30.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:30.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:30.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:30.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:30.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:30.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:30.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:30.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:30.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:31.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19217210 2024-04-25T08:13:31.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19217210 '!=' null 2024-04-25T08:13:31.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:31.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:31.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:31.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:31.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:31.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:31.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:31.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:31.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:31.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:31.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:31.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:31.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:31.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:31.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:32.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:32.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:32.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:32.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:32.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:32.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:32.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:32.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:32.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:33.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27078993 2024-04-25T08:13:33.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27078993 '!=' null 2024-04-25T08:13:33.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:33.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:33.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:33.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:33.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:33.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:33.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:33.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:33.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:33.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:33.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:34.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:34.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:34.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:34.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:35.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:35.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:35.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:35.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:35.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:35.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:35.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:37.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470430 2024-04-25T08:13:37.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470430 '!=' null 2024-04-25T08:13:37.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:37.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:37.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:37.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:37.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:37.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:37.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:37.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:37.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:37.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:37.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:37.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:37.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:37.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:37.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:38.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:38.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:38.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:38.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:38.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:38.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:38.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:38.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:38.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:39.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27953002 2024-04-25T08:13:39.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27953002 '!=' null 2024-04-25T08:13:39.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:39.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:39.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:40.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:40.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:40.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:40.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:40.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:40.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:40.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:40.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:40.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:40.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:40.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:40.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:41.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:41.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:41.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:41.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:41.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:41.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:41.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:41.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:41.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:42.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344485 2024-04-25T08:13:42.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344485 '!=' null 2024-04-25T08:13:42.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:42.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:42.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:42.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:42.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:42.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:42.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:42.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:42.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:42.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:42.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:42.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:42.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:42.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:42.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:43.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:43.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:43.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:43.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:43.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:43.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:43.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:43.718 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:43.719 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:44.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080263 2024-04-25T08:13:44.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080263 '!=' null 2024-04-25T08:13:44.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:44.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:44.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:44.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:44.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:44.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:44.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:44.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:44.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:44.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:44.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:45.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:45.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:45.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:45.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:46.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:46.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:46.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:46.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:46.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:46.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:46.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:46.271 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:46.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:47.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218086 2024-04-25T08:13:47.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218086 '!=' null 2024-04-25T08:13:47.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:47.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:47.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:47.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:47.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:47.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:47.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:47.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:47.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:47.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:47.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:47.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:47.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:47.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:47.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:48.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:48.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:48.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:48.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:48.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:48.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:48.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:48.918 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:48.918 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:49.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827034 2024-04-25T08:13:49.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827034 '!=' null 2024-04-25T08:13:49.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:49.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:49.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:50.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:50.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:50.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:50.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:50.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:50.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:50.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:50.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:50.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:50.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:50.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:50.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:51.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:51.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:51.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:51.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:51.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:51.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:51.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:51.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:51.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:51.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344502 2024-04-25T08:13:51.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344502 '!=' null 2024-04-25T08:13:51.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:51.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:51.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:51.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:51.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:51.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:51.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:51.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:51.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:51.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:51.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:52.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:52.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:52.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:52.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:53.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:53.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:53.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:53.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:53.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:53.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:53.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:53.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:53.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:54.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080693 2024-04-25T08:13:54.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080693 '!=' null 2024-04-25T08:13:54.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:54.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:54.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:54.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:54.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:54.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:54.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:54.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:54.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:54.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:54.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:55.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:55.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:55.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:55.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:56.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:56.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:56.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:56.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:56.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:56.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:56.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:56.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:56.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:57.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344984 2024-04-25T08:13:57.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344984 '!=' null 2024-04-25T08:13:57.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:57.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:57.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:13:57.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:13:57.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:13:57.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:13:57.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:13:57.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:13:57.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:13:57.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:13:57.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:13:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:13:58.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:13:58.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:13:58.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:13:58.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:13:58.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:13:58.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:13:58.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:13:58.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:13:58.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:13:58.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:13:58.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:13:58.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:13:59.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954396 2024-04-25T08:13:59.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954396 '!=' null 2024-04-25T08:13:59.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:13:59.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:13:59.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:00.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:00.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:00.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:00.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:00.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:00.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:00.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:00.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:00.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:00.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:00.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:00.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:01.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:01.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:01.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:01.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:01.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:01.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:01.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:01.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:01.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:02.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18345075 2024-04-25T08:14:02.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18345075 '!=' null 2024-04-25T08:14:02.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:02.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:02.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:02.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:02.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:02.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:02.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:02.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:02.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:02.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:02.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:03.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:03.466 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:03.466 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:03.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:04.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:04.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:04.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:04.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:04.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:04.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:04.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:04.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:04.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:05.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344579 2024-04-25T08:14:05.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344579 '!=' null 2024-04-25T08:14:05.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:05.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:05.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:05.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:05.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:05.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:05.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:05.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:05.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:05.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:05.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:06.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:06.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:06.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:06.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:07.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:07.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:07.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:07.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:07.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:07.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:07.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:07.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:07.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344388 2024-04-25T08:14:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344388 '!=' null 2024-04-25T08:14:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:08.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:08.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:08.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:08.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:08.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:08.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:08.326 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:08.326 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:08.326 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:08.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:08.968 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:08.968 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:08.968 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:10.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:10.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:10.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:10.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:10.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:10.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:10.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:10.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:10.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:11.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218003 2024-04-25T08:14:11.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218003 '!=' null 2024-04-25T08:14:11.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:11.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:11.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:11.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:11.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:11.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:11.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:11.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:11.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:11.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:11.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:11.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:11.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:11.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:11.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:12.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:12.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:12.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:12.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:12.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:12.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:12.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:12.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:12.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:13.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954075 2024-04-25T08:14:13.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954075 '!=' null 2024-04-25T08:14:13.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:13.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:13.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:13.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:13.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:13.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:13.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:13.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:13.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:13.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:13.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:13.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:13.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:13.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:13.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:14.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:14.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:14.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:14.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:14.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:14.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:14.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:14.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:14.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:15.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218406 2024-04-25T08:14:15.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218406 '!=' null 2024-04-25T08:14:15.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:15.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:15.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:15.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:15.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:15.664 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:16.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:16.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:16.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:16.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:17.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:17.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:18.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218399 2024-04-25T08:14:18.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218399 '!=' null 2024-04-25T08:14:18.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:18.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:18.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:18.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:18.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:18.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:18.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:18.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:18.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:18.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:18.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:19.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:19.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:19.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:19.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:20.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:20.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:20.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:20.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:20.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:20.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:20.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:20.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:20.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:21.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28827471 2024-04-25T08:14:21.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28827471 '!=' null 2024-04-25T08:14:21.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:21.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:21.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:21.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:21.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:21.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:21.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:21.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:21.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:21.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:21.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:22.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:22.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:22.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:22.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:23.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:23.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:23.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:24.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27953896 2024-04-25T08:14:24.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27953896 '!=' null 2024-04-25T08:14:24.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:24.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:24.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:24.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:24.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:24.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:24.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:24.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:24.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:24.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:24.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:25.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:25.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:25.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:25.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:26.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:26.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:26.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:27.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19217967 2024-04-25T08:14:27.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19217967 '!=' null 2024-04-25T08:14:27.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:27.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:27.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:27.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:27.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:27.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:27.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:27.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:27.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:27.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:27.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:28.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:28.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:28.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:28.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:29.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:29.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:29.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:29.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:29.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:29.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:29.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:29.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:29.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:30.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27080482 2024-04-25T08:14:30.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27080482 '!=' null 2024-04-25T08:14:30.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:30.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:30.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:30.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:30.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:30.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:30.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:30.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:30.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:30.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:30.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:31.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:31.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:31.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:31.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:31.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:31.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:31.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:31.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:31.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:31.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:31.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:31.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:31.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:32.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471313 2024-04-25T08:14:32.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471313 '!=' null 2024-04-25T08:14:32.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:32.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:32.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:32.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:32.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:32.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:32.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:32.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:32.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:32.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:32.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:33.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:33.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:33.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:33.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:34.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:34.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:34.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:34.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:34.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:34.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:34.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:34.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:34.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:35.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597831 2024-04-25T08:14:35.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597831 '!=' null 2024-04-25T08:14:35.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:35.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:35.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:35.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:35.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:35.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:35.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:35.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:35.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:35.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:35.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:36.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:36.213 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:36.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:36.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:37.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:37.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:37.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:37.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:37.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:37.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:37.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:37.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:37.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:38.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597878 2024-04-25T08:14:38.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597878 '!=' null 2024-04-25T08:14:38.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:38.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:38.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:38.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:38.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:38.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:38.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:38.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:38.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:38.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:38.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:39.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:39.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:39.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:39.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:40.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:40.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:40.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:40.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=26207424 2024-04-25T08:14:40.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 26207424 '!=' null 2024-04-25T08:14:40.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:40.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:40.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:40.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:40.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:40.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:40.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:40.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:40.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:40.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:40.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:41.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:41.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:41.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:41.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:42.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:42.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:42.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:42.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:42.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:42.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:44.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27954499 2024-04-25T08:14:44.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27954499 '!=' null 2024-04-25T08:14:44.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:44.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:44.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:44.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:44.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:44.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:44.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:44.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:44.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:44.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:44.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:45.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:45.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:45.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:45.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:45.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:45.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:45.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:45.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:45.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:45.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:45.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:45.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:45.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:46.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218575 2024-04-25T08:14:46.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218575 '!=' null 2024-04-25T08:14:46.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:46.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:46.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:46.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:46.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:46.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:47.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:47.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:47.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:47.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:47.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:47.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:47.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:47.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:47.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:47.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:47.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:47.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:47.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:48.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218462 2024-04-25T08:14:48.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218462 '!=' null 2024-04-25T08:14:48.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:48.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:48.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:49.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:49.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:49.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:49.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:49.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:49.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:49.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:49.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:49.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:49.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:49.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:49.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:50.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:50.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:50.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:50.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:50.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:50.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:50.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:50.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:50.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:51.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218184 2024-04-25T08:14:51.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218184 '!=' null 2024-04-25T08:14:51.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:51.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:51.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:51.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:51.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:51.387 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:51.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:51.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:51.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:51.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:51.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:51.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:51.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:51.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:51.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:52.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:52.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:52.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:52.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:52.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:52.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:52.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:52.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:52.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:53.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23061734 2024-04-25T08:14:53.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23061734 '!=' null 2024-04-25T08:14:53.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:53.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:53.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:53.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:53.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:53.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:53.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:53.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:53.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:53.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:53.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:53.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:53.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:53.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:53.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:54.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:54.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:54.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:54.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:54.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:54.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:54.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:54.716 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:54.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:55.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=28828330 2024-04-25T08:14:55.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 28828330 '!=' null 2024-04-25T08:14:55.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:55.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:55.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:55.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:55.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:55.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:55.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:55.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:55.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:55.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:55.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:55.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:55.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:55.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:55.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:56.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:56.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:56.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:56.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:56.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:56.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:56.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:56.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:56.592 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:57.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14414106 2024-04-25T08:14:57.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14414106 '!=' null 2024-04-25T08:14:57.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:57.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:57.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:57.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:57.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:57.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:57.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:57.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:57.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:57.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:57.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:14:57.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:14:57.711 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:14:57.711 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:14:57.711 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:14:58.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 2 2024-04-25T08:14:58.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:14:58.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:14:58.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:14:58.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:14:58.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:14:58.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:14:58.719 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:14:58.719 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:14:59.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24897088 2024-04-25T08:14:59.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24897088 '!=' null 2024-04-25T08:14:59.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:14:59.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:14:59.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:14:59.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:14:59.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:14:59.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:14:59.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:14:59.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:14:59.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:14:59.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:14:59.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:00.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:00.152 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:00.152 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:00.152 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:00.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:00.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-04-25T08:15:00.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:15:00.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-04-25T08:15:00.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:00.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:01.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:01.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:01.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:01.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:01.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:01.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:01.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:01.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:01.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:01.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:01.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:01.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:02.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:02.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:02.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:02.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:02.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:02.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:02.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:02.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:02.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16772470 2024-04-25T08:15:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16772470 '!=' null 2024-04-25T08:15:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:03.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:03.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:03.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:03.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:03.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:03.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:03.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:03.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:03.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:04.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:04.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:04.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:04.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:04.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:04.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:04.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:04.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:04.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:04.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:04.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:04.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:04.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:05.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15055428 2024-04-25T08:15:05.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15055428 '!=' null 2024-04-25T08:15:05.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:05.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:05.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:05.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:05.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:05.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:05.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:05.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:05.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:05.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:05.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:06.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:06.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:06.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:06.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:07.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:07.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:07.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13977054 2024-04-25T08:15:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13977054 '!=' null 2024-04-25T08:15:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:08.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:08.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:08.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:08.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:08.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:08.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:08.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:08.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:08.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:08.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:08.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:08.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:09.557 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:09.558 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:10.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839126 2024-04-25T08:15:10.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839126 '!=' null 2024-04-25T08:15:10.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:10.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:10.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:10.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:10.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:10.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:10.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:10.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:10.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:10.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:10.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:11.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:11.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:11.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:11.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:12.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:12.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:12.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:12.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:12.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:12.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:12.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:12.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:12.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:13.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712775 2024-04-25T08:15:13.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712775 '!=' null 2024-04-25T08:15:13.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:13.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:13.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:13.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:13.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:13.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:13.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:13.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:13.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:13.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:13.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:13.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:13.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:13.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:13.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:14.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:14.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:14.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:14.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:14.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:14.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:14.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:14.412 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:14.412 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:15.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724190 2024-04-25T08:15:15.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724190 '!=' null 2024-04-25T08:15:15.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:15.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:15.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:15.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:15.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:15.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:15.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:15.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:15.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:15.330 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:15.330 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:15.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:15.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:15.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:15.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:16.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:16.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:16.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:16.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:16.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:16.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:16.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:16.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:16.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:17.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724163 2024-04-25T08:15:17.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724163 '!=' null 2024-04-25T08:15:17.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:17.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:17.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:17.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:17.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:17.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:17.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:17.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:17.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:17.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:17.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:17.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:17.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:17.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:17.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:18.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:18.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:18.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:18.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:18.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:18.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:18.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:18.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:18.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:19.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459471 2024-04-25T08:15:19.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459471 '!=' null 2024-04-25T08:15:19.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:19.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:19.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:20.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:20.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:20.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:20.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:20.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:20.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:20.093 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:20.093 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:20.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:20.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:20.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:20.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:21.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:21.373 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:21.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:22.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597653 2024-04-25T08:15:22.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597653 '!=' null 2024-04-25T08:15:22.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:22.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:22.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:22.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:22.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:22.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:22.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:22.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:22.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:22.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:22.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:22.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:22.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:22.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:22.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:23.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:23.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:23.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:23.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:23.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:23.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:23.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:23.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:23.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:24.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=23586259 2024-04-25T08:15:24.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 23586259 '!=' null 2024-04-25T08:15:24.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:24.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:24.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:24.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:24.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:24.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:24.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:24.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:24.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:24.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:24.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:24.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:24.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:24.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:24.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:25.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:25.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:25.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:25.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:26.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723967 2024-04-25T08:15:26.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723967 '!=' null 2024-04-25T08:15:26.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:26.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:26.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:26.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:26.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:26.753 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:26.753 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:26.753 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:26.753 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:26.753 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:26.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:27.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:27.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:27.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:27.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:28.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:28.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:28.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:28.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:28.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:28.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:28.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:28.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:28.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:29.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723944 2024-04-25T08:15:29.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723944 '!=' null 2024-04-25T08:15:29.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:29.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:29.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:29.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:29.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:29.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:29.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:29.287 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:29.287 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:29.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:29.706 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:29.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:29.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:30.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:30.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:30.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:30.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:30.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:30.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:30.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:30.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:30.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:31.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712319 2024-04-25T08:15:31.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712319 '!=' null 2024-04-25T08:15:31.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:31.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:31.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:31.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:31.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:31.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:31.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:31.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:31.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:31.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:31.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:31.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:31.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:31.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:31.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:32.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:32.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:33.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838593 2024-04-25T08:15:33.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838593 '!=' null 2024-04-25T08:15:33.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:33.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:33.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:33.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:33.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:33.667 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:33.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:33.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:33.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:33.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:33.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:34.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:34.153 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:34.153 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:34.153 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:34.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:34.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:34.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:34.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:34.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:34.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:34.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:34.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:34.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:35.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850159 2024-04-25T08:15:35.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850159 '!=' null 2024-04-25T08:15:35.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:35.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:35.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:35.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:35.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:35.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:35.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:35.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:35.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:35.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:35.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:35.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:35.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:35.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:35.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:36.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:36.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:36.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:36.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:36.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:36.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:36.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:36.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:36.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:37.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850620 2024-04-25T08:15:37.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850620 '!=' null 2024-04-25T08:15:37.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:37.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:37.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:37.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:37.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:37.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:37.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:37.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:37.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:37.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:37.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:37.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:37.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:37.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:37.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:38.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:38.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:38.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:38.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:38.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:38.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:38.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:38.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:38.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:39.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839277 2024-04-25T08:15:39.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839277 '!=' null 2024-04-25T08:15:39.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:39.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:39.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:39.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:39.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:39.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:39.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:39.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:39.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:39.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:39.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:39.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:39.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:39.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:39.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:40.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:40.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:40.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:40.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:40.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:40.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:40.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:40.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:40.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:40.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850717 2024-04-25T08:15:40.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850717 '!=' null 2024-04-25T08:15:40.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:40.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:40.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:41.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:41.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:41.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:41.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:41.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:41.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:41.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:41.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:41.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:41.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:41.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:41.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:42.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:42.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:42.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:42.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:42.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:42.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:42.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:42.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:42.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:42.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965662 2024-04-25T08:15:42.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965662 '!=' null 2024-04-25T08:15:42.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:42.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:42.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:42.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:42.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:42.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:42.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:42.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:42.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:42.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:42.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:43.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:43.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:43.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:43.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:43.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:43.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:43.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:43.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:43.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:43.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:43.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:43.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:43.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:44.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965662 2024-04-25T08:15:44.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965662 '!=' null 2024-04-25T08:15:44.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:44.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:44.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:44.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:44.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:44.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:44.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:44.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:44.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:44.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:44.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:45.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:45.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:45.087 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:45.087 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:45.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:45.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:45.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:45.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:45.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:45.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:45.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:45.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:45.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:46.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13103512 2024-04-25T08:15:46.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13103512 '!=' null 2024-04-25T08:15:46.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:46.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:46.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:46.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:46.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:46.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:46.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:46.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:46.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:46.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:46.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:46.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:46.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:46.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:46.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:47.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:47.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:47.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:47.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:47.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:47.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:47.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:47.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:47.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:48.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13103534 2024-04-25T08:15:48.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13103534 '!=' null 2024-04-25T08:15:48.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:48.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:48.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:48.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:48.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:48.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:48.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:48.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:48.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:48.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:48.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:48.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:48.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:48.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:48.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:49.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:49.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:49.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:49.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:49.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:49.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:49.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:49.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:49.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:49.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17034443 2024-04-25T08:15:49.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17034443 '!=' null 2024-04-25T08:15:49.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:49.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:49.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:49.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:49.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:49.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:49.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:49.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:49.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:49.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:49.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:50.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:50.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:50.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:50.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:50.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:50.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:50.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:50.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:50.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:50.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:50.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:50.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:50.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7862206 2024-04-25T08:15:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7862206 '!=' null 2024-04-25T08:15:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:51.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:51.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:51.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:51.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:51.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:51.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:51.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:51.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:52.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:15:52.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:52.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:52.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:52.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 2 2024-04-25T08:15:52.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:15:52.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:15:52.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:15:52.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:15:52.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:15:52.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:15:52.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:15:52.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:15:53.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7862265 2024-04-25T08:15:53.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7862265 '!=' null 2024-04-25T08:15:53.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:15:53.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:15:53.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:15:53.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:15:53.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:15:53.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:15:53.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:15:53.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:15:53.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:15:53.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:15:53.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:15:53.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:15:53.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:15:53.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:15:53.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:15:54.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T08:15:54.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:15:54.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:15:54.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:525: TEST_recovery_pool_priority: '[' 0 '!=' 0 ']' 2024-04-25T08:15:54.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:529: TEST_recovery_pool_priority: echo TEST PASSED 2024-04-25T08:15:54.184 INFO:tasks.workunit.client.0.smithi134.stdout:TEST PASSED 2024-04-25T08:15:54.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:532: TEST_recovery_pool_priority: delete_pool test1 2024-04-25T08:15:54.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:15:54.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:15:54.882 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test1' does not exist 2024-04-25T08:15:54.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:533: TEST_recovery_pool_priority: delete_pool test2 2024-04-25T08:15:54.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-04-25T08:15:54.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-04-25T08:15:55.824 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test2' does not exist 2024-04-25T08:15:55.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:534: TEST_recovery_pool_priority: kill_daemons td/osd-recovery-prio 2024-04-25T08:15:55.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:15:55.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:15:55.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:15:55.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:15:55.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:16:01.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:16:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:535: TEST_recovery_pool_priority: return 0 2024-04-25T08:16:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-04-25T08:16:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:16:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:16:01.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:16:01.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:16:01.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:16:01.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:16:01.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:16:01.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:16:01.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:16:01.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:16:01.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:16:01.175 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:16:01.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:16:01.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:16:01.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:16:01.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:16:01.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:16:01.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:16:01.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:16:01.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:16:01.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:16:01.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:16:01.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:16:01.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:01.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:01.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.206040 2024-04-25T08:16:01.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:38: run: for func in $funcs 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:39: run: setup td/osd-recovery-prio 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-prio 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-prio 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:16:01.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:16:01.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:16:01.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:16:01.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:16:01.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:16:01.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:16:01.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:16:01.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:16:01.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:16:01.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:16:01.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:16:01.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:16:01.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:16:01.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:16:01.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:16:01.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:16:01.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:01.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:01.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.206040 2024-04-25T08:16:01.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:16:01.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:16:01.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-prio 2024-04-25T08:16:01.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:16:01.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:01.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:01.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.206040 2024-04-25T08:16:01.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:16:01.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:16:01.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-prio 1' TERM HUP INT 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:40: run: TEST_recovery_priority td/osd-recovery-prio 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:47: TEST_recovery_priority: local dir=td/osd-recovery-prio 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:48: TEST_recovery_priority: local pools=10 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:49: TEST_recovery_priority: local OSDS=5 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:50: TEST_recovery_priority: local max_tries=10 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:52: TEST_recovery_priority: run_mon td/osd-recovery-prio a 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-prio 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-prio/a 2024-04-25T08:16:01.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-prio/a --run-dir=td/osd-recovery-prio 2024-04-25T08:16:01.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:16:01.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:16:01.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:16:01.767 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:16:01.767 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:01.767 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:01.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:01.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-prio/a '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-prio/log --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:16:01.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:16:01.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:16:01.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:16:01.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:16:01.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-mon.a.asok 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:16:01.999 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.206040/ceph-mon.a.asok config get fsid 2024-04-25T08:16:02.118 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:16:02.118 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:16:02.118 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:16:02.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:16:02.120 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:16:02.120 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:16:02.120 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:16:02.120 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:16:02.120 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:16:02.120 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:02.120 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:02.120 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-mon.a.asok 2024-04-25T08:16:02.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:16:02.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.206040/ceph-mon.a.asok config get mon_host 2024-04-25T08:16:02.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:53: TEST_recovery_priority: run_mgr td/osd-recovery-prio x 2024-04-25T08:16:02.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-prio 2024-04-25T08:16:02.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:16:02.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:16:02.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:16:02.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-prio/x 2024-04-25T08:16:02.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:16:03.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:16:03.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:16:03.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:16:03.231 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:16:03.231 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:03.231 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:03.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:03.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:16:03.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-prio/x '--log-file=td/osd-recovery-prio/$name.log' '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --run-dir=td/osd-recovery-prio '--pid-file=td/osd-recovery-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:16:03.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:54: TEST_recovery_priority: export CEPH_ARGS 2024-04-25T08:16:03.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: expr 5 - 1 2024-04-25T08:16:03.275 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: seq 0 4 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 0 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/0 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:16:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/0' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/0/journal' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/0 2024-04-25T08:16:03.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:16:03.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=26214839-db77-4d7a-8a3c-63e0cdc613b6 2024-04-25T08:16:03.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 26214839-db77-4d7a-8a3c-63e0cdc613b6' 2024-04-25T08:16:03.282 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 26214839-db77-4d7a-8a3c-63e0cdc613b6 2024-04-25T08:16:03.283 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:16:03.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBDESpm6P4AEhAAKt69SfbE8flgBOwlHJACIQ== 2024-04-25T08:16:03.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBDESpm6P4AEhAAKt69SfbE8flgBOwlHJACIQ=="}' 2024-04-25T08:16:03.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 26214839-db77-4d7a-8a3c-63e0cdc613b6 -i td/osd-recovery-prio/0/new.json 2024-04-25T08:16:03.673 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:16:03.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/0/new.json 2024-04-25T08:16:03.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBDESpm6P4AEhAAKt69SfbE8flgBOwlHJACIQ== --osd-uuid 26214839-db77-4d7a-8a3c-63e0cdc613b6 2024-04-25T08:16:03.727 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:03.726+0000 7fd285c1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:03.767 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:03.766+0000 7fd285c1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:03.769 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:03.766+0000 7fd285c1f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:03.769 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:03.766+0000 7fd285c1f7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:16:03.769 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:03.766+0000 7fd285c1f7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:16:03.769 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:03.766+0000 7fd285c1f7c0 -1 bluestore(td/osd-recovery-prio/0/block) _read_bdev_label failed to open td/osd-recovery-prio/0/block: (2) No such file or directory 2024-04-25T08:16:03.806 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:03.802+0000 7fd285c1f7c0 -1 bluestore(td/osd-recovery-prio/0) _read_fsid unparsable uuid 2024-04-25T08:16:11.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/0/keyring 2024-04-25T08:16:11.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:16:11.808 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:16:11.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:16:11.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:16:12.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:16:12.400 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:16:12.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:16:12.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/0 --osd-journal=td/osd-recovery-prio/0/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:16:12.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:16:12.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:16:12.434 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:12.430+0000 7f68e08977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:12.436 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:12.434+0000 7f68e08977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:12.438 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:12.434+0000 7f68e08977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:12.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:16:12.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:16:12.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:16:12.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:16:12.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:16:12.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:12.800 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:16:12.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:16:12.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:12.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:16:12.980 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:12.978+0000 7f68e08977c0 -1 Falling back to public interface 2024-04-25T08:16:13.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:14.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:14.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:14.256 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:16:14.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:16:14.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:14.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:16:14.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:15.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:15.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:15.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:16:15.696 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:16:15.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:15.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:16:15.714 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:15.710+0000 7f68e08977c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:16:16.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:17.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:17.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:17.140 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:16:17.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:16:17.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:17.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:16:17.582 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/790733731,v1:127.0.0.1:6803/790733731] [v2:127.0.0.1:6804/790733731,v1:127.0.0.1:6805/790733731] exists,up 26214839-db77-4d7a-8a3c-63e0cdc613b6 2024-04-25T08:16:17.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:16:17.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:16:17.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 1 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/1 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/1' 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/1/journal' 2024-04-25T08:16:17.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:16:17.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:16:17.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:16:17.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:16:17.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:16:17.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:16:17.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:16:17.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:16:17.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:16:17.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/1 2024-04-25T08:16:17.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:16:17.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dc2f023a-6e47-4c0e-81e8-53b1f3c2932d 2024-04-25T08:16:17.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 dc2f023a-6e47-4c0e-81e8-53b1f3c2932d' 2024-04-25T08:16:17.589 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 dc2f023a-6e47-4c0e-81e8-53b1f3c2932d 2024-04-25T08:16:17.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:16:17.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBRESpmUlZKJBAANx5NVjiFywp/1vA54pF9Ww== 2024-04-25T08:16:17.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBRESpmUlZKJBAANx5NVjiFywp/1vA54pF9Ww=="}' 2024-04-25T08:16:17.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dc2f023a-6e47-4c0e-81e8-53b1f3c2932d -i td/osd-recovery-prio/1/new.json 2024-04-25T08:16:18.333 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:16:18.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/1/new.json 2024-04-25T08:16:18.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBRESpmUlZKJBAANx5NVjiFywp/1vA54pF9Ww== --osd-uuid dc2f023a-6e47-4c0e-81e8-53b1f3c2932d 2024-04-25T08:16:18.395 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:18.398+0000 7f8af990e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:18.422 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:18.422+0000 7f8af990e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:18.423 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:18.422+0000 7f8af990e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:18.424 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:18.422+0000 7f8af990e7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:16:18.424 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:18.422+0000 7f8af990e7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:16:18.424 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:18.422+0000 7f8af990e7c0 -1 bluestore(td/osd-recovery-prio/1/block) _read_bdev_label failed to open td/osd-recovery-prio/1/block: (2) No such file or directory 2024-04-25T08:16:18.522 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:18.522+0000 7f8af990e7c0 -1 bluestore(td/osd-recovery-prio/1) _read_fsid unparsable uuid 2024-04-25T08:16:26.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/1/keyring 2024-04-25T08:16:26.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:16:26.656 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:16:26.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:16:26.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:16:27.214 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:16:27.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:16:27.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/1 --osd-journal=td/osd-recovery-prio/1/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:16:27.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:16:27.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:16:27.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:16:27.250 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:27.249+0000 7f0716da17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:27.252 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:27.253+0000 7f0716da17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:27.253 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:27.253+0000 7f0716da17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:16:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:16:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:16:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:16:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:16:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:27.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:16:27.628 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:16:27.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:27.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:16:27.833 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:27.833+0000 7f0716da17c0 -1 Falling back to public interface 2024-04-25T08:16:28.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:29.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:29.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:29.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:16:29.110 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:16:29.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:29.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:16:29.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:30.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:30.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:30.544 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:16:30.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:16:30.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:30.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:16:30.818 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:30.817+0000 7f0716da17c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:16:30.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:31.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:31.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:31.979 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:16:31.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:16:31.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:31.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:16:32.450 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1834093115,v1:127.0.0.1:6811/1834093115] [v2:127.0.0.1:6812/1834093115,v1:127.0.0.1:6813/1834093115] exists,up dc2f023a-6e47-4c0e-81e8-53b1f3c2932d 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 2 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/2 2024-04-25T08:16:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/2' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/2/journal' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:32.455 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:16:32.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/2 2024-04-25T08:16:32.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:16:32.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=13f7c286-dd6f-4e5f-af7c-f116cc2a6572 2024-04-25T08:16:32.457 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 13f7c286-dd6f-4e5f-af7c-f116cc2a6572 2024-04-25T08:16:32.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 13f7c286-dd6f-4e5f-af7c-f116cc2a6572' 2024-04-25T08:16:32.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:16:32.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBgESpmZN1YHBAAvImZojkca0oJsQ2KOaWDvA== 2024-04-25T08:16:32.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBgESpmZN1YHBAAvImZojkca0oJsQ2KOaWDvA=="}' 2024-04-25T08:16:32.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 13f7c286-dd6f-4e5f-af7c-f116cc2a6572 -i td/osd-recovery-prio/2/new.json 2024-04-25T08:16:33.240 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:16:33.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/2/new.json 2024-04-25T08:16:33.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgESpmZN1YHBAAvImZojkca0oJsQ2KOaWDvA== --osd-uuid 13f7c286-dd6f-4e5f-af7c-f116cc2a6572 2024-04-25T08:16:33.302 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:33.301+0000 7fa860f397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:33.329 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:33.329+0000 7fa860f397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:33.333+0000 7fa860f397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:33.333+0000 7fa860f397c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T08:16:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:33.333+0000 7fa860f397c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T08:16:33.332 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:33.333+0000 7fa860f397c0 -1 bluestore(td/osd-recovery-prio/2/block) _read_bdev_label failed to open td/osd-recovery-prio/2/block: (2) No such file or directory 2024-04-25T08:16:33.430 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:33.429+0000 7fa860f397c0 -1 bluestore(td/osd-recovery-prio/2) _read_fsid unparsable uuid 2024-04-25T08:16:41.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/2/keyring 2024-04-25T08:16:41.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:16:41.852 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T08:16:41.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:16:41.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:16:42.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:16:42.433 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T08:16:42.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/2 --osd-journal=td/osd-recovery-prio/2/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:16:42.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:16:42.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:16:42.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:16:42.462 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:42.461+0000 7efcaeeee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:42.464 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:42.465+0000 7efcaeeee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:42.466 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:42.465+0000 7efcaeeee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:42.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:16:42.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:16:42.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:16:42.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:16:42.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:16:42.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:42.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:16:42.835 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:16:42.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:42.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:43.000 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:43.001+0000 7efcaeeee7c0 -1 Falling back to public interface 2024-04-25T08:16:43.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:44.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:44.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:44.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:16:44.247 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:16:44.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:44.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:44.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:45.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:45.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:45.667 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:16:45.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:16:45.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:45.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:45.905 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:45.905+0000 7efcaeeee7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:16:46.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:47.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:47.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:47.116 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:16:47.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:16:47.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:47.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:16:47.569 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3705103183,v1:127.0.0.1:6819/3705103183] [v2:127.0.0.1:6820/3705103183,v1:127.0.0.1:6821/3705103183] exists,up 13f7c286-dd6f-4e5f-af7c-f116cc2a6572 2024-04-25T08:16:47.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:16:47.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 3 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/3 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:16:47.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/3' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/3/journal' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:16:47.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:16:47.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/3 2024-04-25T08:16:47.577 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:16:47.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=679eff36-d48f-4c8b-9307-18a7b987299d 2024-04-25T08:16:47.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 679eff36-d48f-4c8b-9307-18a7b987299d' 2024-04-25T08:16:47.577 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 679eff36-d48f-4c8b-9307-18a7b987299d 2024-04-25T08:16:47.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:16:47.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvESpmCiCRIxAAsLWdDWpt16vwc2lMl3xJOQ== 2024-04-25T08:16:47.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvESpmCiCRIxAAsLWdDWpt16vwc2lMl3xJOQ=="}' 2024-04-25T08:16:47.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 679eff36-d48f-4c8b-9307-18a7b987299d -i td/osd-recovery-prio/3/new.json 2024-04-25T08:16:48.180 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:16:48.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/3/new.json 2024-04-25T08:16:48.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvESpmCiCRIxAAsLWdDWpt16vwc2lMl3xJOQ== --osd-uuid 679eff36-d48f-4c8b-9307-18a7b987299d 2024-04-25T08:16:48.227 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:48.229+0000 7f68e44487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:48.229 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:48.229+0000 7f68e44487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:48.231 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:48.233+0000 7f68e44487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:48.231 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:48.233+0000 7f68e44487c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T08:16:48.231 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:48.233+0000 7f68e44487c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T08:16:48.231 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:48.233+0000 7f68e44487c0 -1 bluestore(td/osd-recovery-prio/3/block) _read_bdev_label failed to open td/osd-recovery-prio/3/block: (2) No such file or directory 2024-04-25T08:16:48.296 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:48.297+0000 7f68e44487c0 -1 bluestore(td/osd-recovery-prio/3) _read_fsid unparsable uuid 2024-04-25T08:16:56.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/3/keyring 2024-04-25T08:16:56.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:16:56.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:16:56.533 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T08:16:56.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:16:57.074 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T08:16:57.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:16:57.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/3 --osd-journal=td/osd-recovery-prio/3/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:16:57.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:16:57.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:16:57.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:16:57.108 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:57.109+0000 7f0f9fd397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:57.110 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:57.113+0000 7f0f9fd397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:57.112 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:57.113+0000 7f0f9fd397c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:16:57.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:16:57.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:16:57.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:16:57.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:16:57.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:16:57.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:57.534 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:16:57.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:16:57.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:57.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:57.652 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:16:57.653+0000 7f0f9fd397c0 -1 Falling back to public interface 2024-04-25T08:16:58.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:16:59.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:16:59.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:16:59.043 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:16:59.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:16:59.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:16:59.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:16:59.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:00.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:00.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:00.474 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:17:00.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:17:00.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:00.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:17:00.905 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:00.905+0000 7f0f9fd397c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:17:00.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:01.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:01.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:01.923 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:17:01.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:17:01.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:01.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:17:02.460 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3565888802,v1:127.0.0.1:6827/3565888802] [v2:127.0.0.1:6828/3565888802,v1:127.0.0.1:6829/3565888802] exists,up 679eff36-d48f-4c8b-9307-18a7b987299d 2024-04-25T08:17:02.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:17:02.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:17:02.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:17:02.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:56: TEST_recovery_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:17:02.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:58: TEST_recovery_priority: run_osd td/osd-recovery-prio 4 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-prio 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-prio/4 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq ' 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-prio/4' 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-prio/4/journal' 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-prio' 2024-04-25T08:17:02.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:17:02.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:17:02.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:17:02.465 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:17:02.465 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:17:02.465 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:17:02.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:17:02.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' 2024-04-25T08:17:02.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:17:02.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:17:02.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:17:02.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-prio/$name.log' 2024-04-25T08:17:02.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-prio/$name.pid' 2024-04-25T08:17:02.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:17:02.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:17:02.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:17:02.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:17:02.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:17:02.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:17:02.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-prio/4 2024-04-25T08:17:02.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:17:02.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=67193b5b-473c-4b24-b38b-3af0ff7c536d 2024-04-25T08:17:02.467 INFO:tasks.workunit.client.0.smithi134.stdout:add osd4 67193b5b-473c-4b24-b38b-3af0ff7c536d 2024-04-25T08:17:02.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 67193b5b-473c-4b24-b38b-3af0ff7c536d' 2024-04-25T08:17:02.468 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:17:02.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB+ESpm2IpcHRAAJfrhaB4MhUTwKifvtUpteQ== 2024-04-25T08:17:02.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB+ESpm2IpcHRAAJfrhaB4MhUTwKifvtUpteQ=="}' 2024-04-25T08:17:02.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 67193b5b-473c-4b24-b38b-3af0ff7c536d -i td/osd-recovery-prio/4/new.json 2024-04-25T08:17:03.285 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:17:03.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-prio/4/new.json 2024-04-25T08:17:03.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+ESpm2IpcHRAAJfrhaB4MhUTwKifvtUpteQ== --osd-uuid 67193b5b-473c-4b24-b38b-3af0ff7c536d 2024-04-25T08:17:03.342 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:03.341+0000 7f4da2f5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:17:03.345 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:03.345+0000 7f4da2f5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:17:03.346 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:03.345+0000 7f4da2f5f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:17:03.347 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:03.349+0000 7f4da2f5f7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T08:17:03.347 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:03.349+0000 7f4da2f5f7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T08:17:03.347 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:03.349+0000 7f4da2f5f7c0 -1 bluestore(td/osd-recovery-prio/4/block) _read_bdev_label failed to open td/osd-recovery-prio/4/block: (2) No such file or directory 2024-04-25T08:17:03.466 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:03.465+0000 7f4da2f5f7c0 -1 bluestore(td/osd-recovery-prio/4) _read_fsid unparsable uuid 2024-04-25T08:17:11.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-prio/4/keyring 2024-04-25T08:17:11.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:17:11.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:17:11.732 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd4 key to auth repository 2024-04-25T08:17:11.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:17:12.305 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.4 2024-04-25T08:17:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:17:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=9016462e-7c09-474f-8151-12178d1f15c8 --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd-op-queue=wpq --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-prio/4 --osd-journal=td/osd-recovery-prio/4/journal --chdir= --run-dir=td/osd-recovery-prio '--admin-socket=/tmp/ceph-asok.206040/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-prio/$name.log' '--pid-file=td/osd-recovery-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:17:12.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:17:12.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:17:12.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:17:12.339 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:12.340+0000 7f48538387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:17:12.342 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:12.340+0000 7f48538387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:17:12.343 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:12.344+0000 7f48538387c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:17:12.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:17:12.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:17:12.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:17:12.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:17:12.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:17:12.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:12.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:17:12.722 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:17:12.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:12.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:17:12.925 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:12.924+0000 7f48538387c0 -1 Falling back to public interface 2024-04-25T08:17:13.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:14.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:14.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:14.132 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:17:14.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:17:14.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:14.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:17:14.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:15.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:15.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:15.549 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:17:15.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:17:15.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:15.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:17:15.762 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:17:15.760+0000 7f48538387c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:17:16.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:17:17.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:17:17.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:17:17.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:17:17.074 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:17:17.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:17:17.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:17:17.484 INFO:tasks.workunit.client.0.smithi134.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3945226431,v1:127.0.0.1:6835/3945226431] [v2:127.0.0.1:6836/3945226431,v1:127.0.0.1:6837/3945226431] exists,up 67193b5b-473c-4b24-b38b-3af0ff7c536d 2024-04-25T08:17:17.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:17:17.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:17:17.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:17:17.485 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: seq 1 10 2024-04-25T08:17:17.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:17.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test1 1 1 2024-04-25T08:17:17.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:17:18.022 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test1' already exists 2024-04-25T08:17:18.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test1 size 2 2024-04-25T08:17:20.213 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 2 2024-04-25T08:17:20.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:20.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test2 1 1 2024-04-25T08:17:20.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test2 1 1 2024-04-25T08:17:21.092 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test2' already exists 2024-04-25T08:17:21.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:22.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test2 size 2 2024-04-25T08:17:23.559 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 2 size to 2 2024-04-25T08:17:23.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:23.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test3 1 1 2024-04-25T08:17:23.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test3 1 1 2024-04-25T08:17:24.496 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test3' already exists 2024-04-25T08:17:24.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:25.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-04-25T08:17:26.978 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 3 size to 2 2024-04-25T08:17:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test4 1 1 2024-04-25T08:17:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test4 1 1 2024-04-25T08:17:28.053 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test4' already exists 2024-04-25T08:17:28.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:29.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test4 size 2 2024-04-25T08:17:30.111 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 4 size to 2 2024-04-25T08:17:30.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:30.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test5 1 1 2024-04-25T08:17:30.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test5 1 1 2024-04-25T08:17:31.173 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test5' already exists 2024-04-25T08:17:31.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:32.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test5 size 2 2024-04-25T08:17:33.584 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 5 size to 2 2024-04-25T08:17:33.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:33.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test6 1 1 2024-04-25T08:17:33.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test6 1 1 2024-04-25T08:17:34.704 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test6' already exists 2024-04-25T08:17:34.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:35.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test6 size 2 2024-04-25T08:17:36.654 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 6 size to 2 2024-04-25T08:17:36.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:36.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test7 1 1 2024-04-25T08:17:36.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test7 1 1 2024-04-25T08:17:37.566 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test7' already exists 2024-04-25T08:17:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:38.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-04-25T08:17:39.942 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 7 size to 2 2024-04-25T08:17:39.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:39.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test8 1 1 2024-04-25T08:17:39.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test8 1 1 2024-04-25T08:17:41.100 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test8' already exists 2024-04-25T08:17:41.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:42.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test8 size 2 2024-04-25T08:17:43.526 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 8 size to 2 2024-04-25T08:17:43.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:43.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test9 1 1 2024-04-25T08:17:43.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test9 1 1 2024-04-25T08:17:44.538 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test9' already exists 2024-04-25T08:17:44.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:45.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test9 size 2 2024-04-25T08:17:47.093 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 9 size to 2 2024-04-25T08:17:47.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:61: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:17:47.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:63: TEST_recovery_priority: create_pool test10 1 1 2024-04-25T08:17:47.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test10 1 1 2024-04-25T08:17:47.809 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test10' already exists 2024-04-25T08:17:47.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:17:48.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:64: TEST_recovery_priority: ceph osd pool set test10 size 2 2024-04-25T08:17:50.231 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 10 size to 2 2024-04-25T08:17:50.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:66: TEST_recovery_priority: sleep 5 2024-04-25T08:17:55.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:68: TEST_recovery_priority: wait_for_clean 2024-04-25T08:17:55.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:17:55.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:17:55.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:17:55.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:17:55.259 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:17:55.260 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:17:55.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:17:55.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:17:55.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:17:55.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:17:55.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:17:55.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:17:55.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:17:55.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:17:55.381 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:17:55.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:17:55.820 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:17:55.821 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:17:55.821 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:17:55.821 INFO:tasks.workunit.client.0.smithi134.stderr:4' 2024-04-25T08:17:55.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:17:55.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:17:55.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:17:56.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836502 2024-04-25T08:17:56.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836502 2024-04-25T08:17:56.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502' 2024-04-25T08:17:56.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:17:56.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:17:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672979 2024-04-25T08:17:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672979 2024-04-25T08:17:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672979' 2024-04-25T08:17:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:17:56.220 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:17:56.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509456 2024-04-25T08:17:56.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509456 2024-04-25T08:17:56.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672979 2-64424509456' 2024-04-25T08:17:56.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:17:56.373 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:17:56.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378637 2024-04-25T08:17:56.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378637 2024-04-25T08:17:56.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672979 2-64424509456 3-81604378637' 2024-04-25T08:17:56.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:17:56.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:17:56.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215114 2024-04-25T08:17:56.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215114 2024-04-25T08:17:56.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836502 1-42949672979 2-64424509456 3-81604378637 4-103079215114' 2024-04-25T08:17:56.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:17:56.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836502 2024-04-25T08:17:56.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:17:56.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:17:56.683 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836502 2024-04-25T08:17:56.683 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:17:56.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836502 2024-04-25T08:17:56.684 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836502 2024-04-25T08:17:56.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2024-04-25T08:17:56.684 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:17:57.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836500 -lt 21474836502 2024-04-25T08:17:57.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:17:58.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:17:58.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:17:58.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836503 -lt 21474836502 2024-04-25T08:17:58.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:17:58.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672979 2024-04-25T08:17:58.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:17:58.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:17:58.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:17:58.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672979 2024-04-25T08:17:58.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672979 2024-04-25T08:17:58.539 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672979 2024-04-25T08:17:58.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-04-25T08:17:58.539 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:17:58.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672980 -lt 42949672979 2024-04-25T08:17:58.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:17:58.934 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509456 2024-04-25T08:17:58.934 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:17:58.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:17:58.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-04-25T08:17:58.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:17:58.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509456 2024-04-25T08:17:58.937 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509456 2024-04-25T08:17:58.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-04-25T08:17:58.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:17:59.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509456 2024-04-25T08:17:59.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:17:59.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378637 2024-04-25T08:17:59.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:17:59.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:17:59.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378637 2024-04-25T08:17:59.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:17:59.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378637 2024-04-25T08:17:59.401 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378637 2024-04-25T08:17:59.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378637' 2024-04-25T08:17:59.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:17:59.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378638 -lt 81604378637 2024-04-25T08:17:59.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:17:59.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215114 2024-04-25T08:17:59.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:17:59.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:17:59.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215114 2024-04-25T08:17:59.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:17:59.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215114 2024-04-25T08:17:59.860 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215114 2024-04-25T08:17:59.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215114' 2024-04-25T08:17:59.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:18:00.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215115 -lt 103079215114 2024-04-25T08:18:00.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:18:00.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:00.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:00.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 10 == 0 2024-04-25T08:18:00.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:00.883 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:00.883 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:00.884 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:00.884 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:00.884 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:00.884 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:01.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=10 2024-04-25T08:18:01.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:01.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:01.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 10 = 10 2024-04-25T08:18:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:18:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:18:01.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:70: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:18:02.397 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:18:02.397 INFO:tasks.workunit.client.0.smithi134.stdout:10.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:51.119626+0000 0'0 75:20 [2,1] 2 [2,1] 2 0'0 2024-04-25T08:17:47.500092+0000 0'0 2024-04-25T08:17:47.500092+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:15:15.471750+0000 0 0 2024-04-25T08:18:02.397 INFO:tasks.workunit.client.0.smithi134.stdout:9.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:48.321381+0000 0'0 75:29 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:17:44.087952+0000 0'0 2024-04-25T08:17:44.087952+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:07:01.063200+0000 0 0 2024-04-25T08:18:02.397 INFO:tasks.workunit.client.0.smithi134.stdout:8.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:44.675658+0000 0'0 75:37 [3,4] 3 [3,4] 3 0'0 2024-04-25T08:17:40.672862+0000 0'0 2024-04-25T08:17:40.672862+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:05:48.526974+0000 0 0 2024-04-25T08:18:02.397 INFO:tasks.workunit.client.0.smithi134.stdout:7.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:41.603908+0000 0'0 75:45 [1,4] 1 [1,4] 1 0'0 2024-04-25T08:17:37.191617+0000 0'0 2024-04-25T08:17:37.191617+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:52:51.897262+0000 0 0 2024-04-25T08:18:02.397 INFO:tasks.workunit.client.0.smithi134.stdout:6.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:38.825653+0000 0'0 75:49 [4,2] 4 [4,2] 4 0'0 2024-04-25T08:17:34.314949+0000 0'0 2024-04-25T08:17:34.314949+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:35:17.298958+0000 0 0 2024-04-25T08:18:02.397 INFO:tasks.workunit.client.0.smithi134.stdout:5.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:35.020112+0000 0'0 75:63 [3,4] 3 [3,4] 3 0'0 2024-04-25T08:17:30.712049+0000 0'0 2024-04-25T08:17:30.712049+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:23:56.992615+0000 0 0 2024-04-25T08:18:02.397 INFO:tasks.workunit.client.0.smithi134.stdout:4.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:32.549207+0000 0'0 75:71 [4,0] 4 [4,0] 4 0'0 2024-04-25T08:17:27.578333+0000 0'0 2024-04-25T08:17:27.578333+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:47:25.017961+0000 0 0 2024-04-25T08:18:02.397 INFO:tasks.workunit.client.0.smithi134.stdout:3.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:28.575976+0000 0'0 75:84 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:17:23.969579+0000 0'0 2024-04-25T08:17:23.969579+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:53:10.867901+0000 0 0 2024-04-25T08:18:02.398 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:24.090725+0000 0'0 75:91 [3,1] 3 [3,1] 3 0'0 2024-04-25T08:17:20.716190+0000 0'0 2024-04-25T08:17:20.716190+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:01:51.143974+0000 0 0 2024-04-25T08:18:02.398 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T08:17:21.386899+0000 0'0 75:102 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:17:17.811742+0000 0'0 2024-04-25T08:17:17.811742+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:20:36.850081+0000 0 0 2024-04-25T08:18:02.398 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:18:02.398 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:18:02.398 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:74: TEST_recovery_priority: local PG1 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:75: TEST_recovery_priority: local POOLNUM1 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:76: TEST_recovery_priority: local pool1 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:77: TEST_recovery_priority: local chk_osd1_1 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:78: TEST_recovery_priority: local chk_osd1_2 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:80: TEST_recovery_priority: local PG2 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:81: TEST_recovery_priority: local POOLNUM2 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:82: TEST_recovery_priority: local pool2 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:83: TEST_recovery_priority: local chk_osd2 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:85: TEST_recovery_priority: local PG3 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:86: TEST_recovery_priority: local POOLNUM3 2024-04-25T08:18:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:87: TEST_recovery_priority: local pool3 2024-04-25T08:18:02.414 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: seq 1 10 2024-04-25T08:18:02.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:02.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 1.0 --format=json 2024-04-25T08:18:02.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:18:02.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:18:02.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T08:18:02.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:18:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-04-25T08:18:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z '' ']' 2024-04-25T08:18:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:96: TEST_recovery_priority: PG1=1.0 2024-04-25T08:18:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:97: TEST_recovery_priority: POOLNUM1=1 2024-04-25T08:18:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:98: TEST_recovery_priority: pool1=test1 2024-04-25T08:18:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:99: TEST_recovery_priority: chk_osd1_1=1 2024-04-25T08:18:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:100: TEST_recovery_priority: chk_osd1_2=0 2024-04-25T08:18:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:02.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 2.0 --format=json 2024-04-25T08:18:02.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:18:03.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:18:03.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-04-25T08:18:03.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:18:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=1 2024-04-25T08:18:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:18:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2024-04-25T08:18:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2024-04-25T08:18:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:03.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 3.0 --format=json 2024-04-25T08:18:03.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:18:03.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:18:03.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T08:18:03.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:18:03.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-04-25T08:18:03.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:18:03.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2024-04-25T08:18:03.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:103: TEST_recovery_priority: PG2=3.0 2024-04-25T08:18:03.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:104: TEST_recovery_priority: POOLNUM2=3 2024-04-25T08:18:03.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:105: TEST_recovery_priority: pool2=test3 2024-04-25T08:18:03.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:106: TEST_recovery_priority: chk_osd2=2 2024-04-25T08:18:03.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:03.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 4.0 --format=json 2024-04-25T08:18:03.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:18:04.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:18:04.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-04-25T08:18:04.123 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:18:04.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=0 2024-04-25T08:18:04.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:18:04.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2024-04-25T08:18:04.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2024-04-25T08:18:04.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:04.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 5.0 --format=json 2024-04-25T08:18:04.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:18:04.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:18:04.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=3 2024-04-25T08:18:04.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:18:04.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-04-25T08:18:04.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:18:04.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2024-04-25T08:18:04.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-04-25T08:18:04.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:04.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 6.0 --format=json 2024-04-25T08:18:04.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:18:04.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:18:04.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=4 2024-04-25T08:18:04.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:18:04.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=2 2024-04-25T08:18:04.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:18:04.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2024-04-25T08:18:04.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2024-04-25T08:18:04.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:89: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:04.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: ceph pg map 7.0 --format=json 2024-04-25T08:18:04.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:91: TEST_recovery_priority: jq '.acting[]' 2024-04-25T08:18:05.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: head -1 td/osd-recovery-prio/acting 2024-04-25T08:18:05.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:92: TEST_recovery_priority: local test_osd1=1 2024-04-25T08:18:05.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: tail -1 td/osd-recovery-prio/acting 2024-04-25T08:18:05.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:93: TEST_recovery_priority: local test_osd2=4 2024-04-25T08:18:05.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:94: TEST_recovery_priority: '[' -z 1.0 ']' 2024-04-25T08:18:05.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:101: TEST_recovery_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2024-04-25T08:18:05.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:107: TEST_recovery_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2024-04-25T08:18:05.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:109: TEST_recovery_priority: PG3=7.0 2024-04-25T08:18:05.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:110: TEST_recovery_priority: POOLNUM3=7 2024-04-25T08:18:05.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:111: TEST_recovery_priority: pool3=test7 2024-04-25T08:18:05.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:112: TEST_recovery_priority: break 2024-04-25T08:18:05.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:115: TEST_recovery_priority: rm -f td/osd-recovery-prio/acting 2024-04-25T08:18:05.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:117: TEST_recovery_priority: '[' test3 = '' -o pool3 = '' ']' 2024-04-25T08:18:05.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: seq 1 10 2024-04-25T08:18:05.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:05.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2024-04-25T08:18:05.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:05.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2024-04-25T08:18:05.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test2 2024-04-25T08:18:05.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test2 2024-04-25T08:18:05.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2024-04-25T08:18:05.996 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test2' does not exist 2024-04-25T08:18:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2024-04-25T08:18:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2024-04-25T08:18:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test4 2024-04-25T08:18:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test4 2024-04-25T08:18:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2024-04-25T08:18:06.706 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test4' does not exist 2024-04-25T08:18:06.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:06.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2024-04-25T08:18:06.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test5 2024-04-25T08:18:06.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test5 2024-04-25T08:18:06.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2024-04-25T08:18:07.610 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test5' does not exist 2024-04-25T08:18:07.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:07.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2024-04-25T08:18:07.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test6 2024-04-25T08:18:07.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test6 2024-04-25T08:18:07.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2024-04-25T08:18:08.517 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test6' does not exist 2024-04-25T08:18:08.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:08.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2024-04-25T08:18:08.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:08.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2024-04-25T08:18:08.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test8 2024-04-25T08:18:08.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test8 2024-04-25T08:18:08.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2024-04-25T08:18:09.113 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test8' does not exist 2024-04-25T08:18:09.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:09.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2024-04-25T08:18:09.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test9 2024-04-25T08:18:09.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test9 2024-04-25T08:18:09.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2024-04-25T08:18:09.840 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test9' does not exist 2024-04-25T08:18:09.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:123: TEST_recovery_priority: for p in $(seq 1 $pools) 2024-04-25T08:18:09.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:125: TEST_recovery_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2024-04-25T08:18:09.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:127: TEST_recovery_priority: delete_pool test10 2024-04-25T08:18:09.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test10 2024-04-25T08:18:09.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2024-04-25T08:18:10.948 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test10' does not exist 2024-04-25T08:18:10.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:131: TEST_recovery_priority: ceph osd pool set test3 size 1 --yes-i-really-mean-it 2024-04-25T08:18:11.860 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 3 size to 1 2024-04-25T08:18:11.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:132: TEST_recovery_priority: ceph osd pool set test7 size 1 --yes-i-really-mean-it 2024-04-25T08:18:12.876 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 7 size to 1 2024-04-25T08:18:12.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:133: TEST_recovery_priority: wait_for_clean 2024-04-25T08:18:12.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:18:12.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:18:12.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:18:12.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:18:12.901 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:18:12.901 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:18:12.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:18:12.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:18:12.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:18:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:18:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:18:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:18:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:18:13.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:18:13.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:18:13.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:18:13.773 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:18:13.773 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:18:13.773 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:18:13.773 INFO:tasks.workunit.client.0.smithi134.stderr:4' 2024-04-25T08:18:13.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:18:13.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:13.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:18:14.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836508 2024-04-25T08:18:14.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836508 2024-04-25T08:18:14.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508' 2024-04-25T08:18:14.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:14.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:18:14.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-04-25T08:18:14.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-04-25T08:18:14.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672985' 2024-04-25T08:18:14.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:14.337 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:18:14.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509462 2024-04-25T08:18:14.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509462 2024-04-25T08:18:14.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672985 2-64424509462' 2024-04-25T08:18:14.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:14.598 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:18:14.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378643 2024-04-25T08:18:14.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378643 2024-04-25T08:18:14.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672985 2-64424509462 3-81604378643' 2024-04-25T08:18:14.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:18:14.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:18:14.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215120 2024-04-25T08:18:14.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215120 2024-04-25T08:18:14.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836508 1-42949672985 2-64424509462 3-81604378643 4-103079215120' 2024-04-25T08:18:14.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:14.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836508 2024-04-25T08:18:14.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:14.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:18:14.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836508 2024-04-25T08:18:14.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:14.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836508 2024-04-25T08:18:14.933 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836508 2024-04-25T08:18:14.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836508' 2024-04-25T08:18:14.934 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:18:15.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836506 -lt 21474836508 2024-04-25T08:18:15.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:18:16.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:18:16.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:18:16.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836508 2024-04-25T08:18:16.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:16.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-04-25T08:18:16.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:16.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:18:16.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-04-25T08:18:16.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:16.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-04-25T08:18:16.785 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672985 2024-04-25T08:18:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-25T08:18:16.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:18:17.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-04-25T08:18:17.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:17.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509462 2024-04-25T08:18:17.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:17.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:18:17.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509462 2024-04-25T08:18:17.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:17.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509462 2024-04-25T08:18:17.289 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509462 2024-04-25T08:18:17.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509462' 2024-04-25T08:18:17.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:18:17.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509462 -lt 64424509462 2024-04-25T08:18:17.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:17.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378643 2024-04-25T08:18:17.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:17.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:18:17.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378643 2024-04-25T08:18:17.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:17.752 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378643 2024-04-25T08:18:17.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378643 2024-04-25T08:18:17.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378643' 2024-04-25T08:18:17.753 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:18:18.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378644 -lt 81604378643 2024-04-25T08:18:18.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:18:18.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215120 2024-04-25T08:18:18.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:18:18.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:18:18.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215120 2024-04-25T08:18:18.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:18:18.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215120 2024-04-25T08:18:18.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215120' 2024-04-25T08:18:18.240 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215120 2024-04-25T08:18:18.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:18:18.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215121 -lt 103079215120 2024-04-25T08:18:18.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:18:18.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:18.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:19.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T08:18:19.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:18:19.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:18:19.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:18:19.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:18:19.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:18:19.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:18:19.208 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:18:19.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:18:19.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:18:19.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:18:19.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:18:20.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:18:20.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:18:20.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:18:20.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:135: TEST_recovery_priority: dd if=/dev/urandom of=td/osd-recovery-prio/data bs=1M count=10 2024-04-25T08:18:20.200 INFO:tasks.workunit.client.0.smithi134.stderr:10+0 records in 2024-04-25T08:18:20.200 INFO:tasks.workunit.client.0.smithi134.stderr:10+0 records out 2024-04-25T08:18:20.200 INFO:tasks.workunit.client.0.smithi134.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0389283 s, 269 MB/s 2024-04-25T08:18:20.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:136: TEST_recovery_priority: p=1 2024-04-25T08:18:20.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T08:18:20.201 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T08:18:20.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:20.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj1-p1 td/osd-recovery-prio/data 2024-04-25T08:18:21.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:21.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj2-p1 td/osd-recovery-prio/data 2024-04-25T08:18:22.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:22.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj3-p1 td/osd-recovery-prio/data 2024-04-25T08:18:23.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:23.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj4-p1 td/osd-recovery-prio/data 2024-04-25T08:18:24.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:24.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj5-p1 td/osd-recovery-prio/data 2024-04-25T08:18:25.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:25.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj6-p1 td/osd-recovery-prio/data 2024-04-25T08:18:26.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:26.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj7-p1 td/osd-recovery-prio/data 2024-04-25T08:18:27.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:27.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj8-p1 td/osd-recovery-prio/data 2024-04-25T08:18:28.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:28.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj9-p1 td/osd-recovery-prio/data 2024-04-25T08:18:29.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:29.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj10-p1 td/osd-recovery-prio/data 2024-04-25T08:18:30.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:30.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj11-p1 td/osd-recovery-prio/data 2024-04-25T08:18:31.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:31.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj12-p1 td/osd-recovery-prio/data 2024-04-25T08:18:32.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:32.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj13-p1 td/osd-recovery-prio/data 2024-04-25T08:18:33.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:33.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj14-p1 td/osd-recovery-prio/data 2024-04-25T08:18:34.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:34.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj15-p1 td/osd-recovery-prio/data 2024-04-25T08:18:35.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:35.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj16-p1 td/osd-recovery-prio/data 2024-04-25T08:18:36.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:36.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj17-p1 td/osd-recovery-prio/data 2024-04-25T08:18:37.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:37.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj18-p1 td/osd-recovery-prio/data 2024-04-25T08:18:38.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:38.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj19-p1 td/osd-recovery-prio/data 2024-04-25T08:18:39.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:39.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj20-p1 td/osd-recovery-prio/data 2024-04-25T08:18:40.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:40.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj21-p1 td/osd-recovery-prio/data 2024-04-25T08:18:41.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:41.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj22-p1 td/osd-recovery-prio/data 2024-04-25T08:18:42.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:42.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj23-p1 td/osd-recovery-prio/data 2024-04-25T08:18:43.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:43.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj24-p1 td/osd-recovery-prio/data 2024-04-25T08:18:44.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:44.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj25-p1 td/osd-recovery-prio/data 2024-04-25T08:18:45.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:45.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj26-p1 td/osd-recovery-prio/data 2024-04-25T08:18:46.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:46.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj27-p1 td/osd-recovery-prio/data 2024-04-25T08:18:47.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:47.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj28-p1 td/osd-recovery-prio/data 2024-04-25T08:18:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj29-p1 td/osd-recovery-prio/data 2024-04-25T08:18:49.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:49.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj30-p1 td/osd-recovery-prio/data 2024-04-25T08:18:50.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:50.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj31-p1 td/osd-recovery-prio/data 2024-04-25T08:18:51.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:51.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj32-p1 td/osd-recovery-prio/data 2024-04-25T08:18:52.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:52.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj33-p1 td/osd-recovery-prio/data 2024-04-25T08:18:53.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:53.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj34-p1 td/osd-recovery-prio/data 2024-04-25T08:18:54.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:54.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj35-p1 td/osd-recovery-prio/data 2024-04-25T08:18:55.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:55.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj36-p1 td/osd-recovery-prio/data 2024-04-25T08:18:56.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:56.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj37-p1 td/osd-recovery-prio/data 2024-04-25T08:18:57.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:57.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj38-p1 td/osd-recovery-prio/data 2024-04-25T08:18:58.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:58.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj39-p1 td/osd-recovery-prio/data 2024-04-25T08:18:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:18:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj40-p1 td/osd-recovery-prio/data 2024-04-25T08:19:00.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:00.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj41-p1 td/osd-recovery-prio/data 2024-04-25T08:19:01.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:01.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj42-p1 td/osd-recovery-prio/data 2024-04-25T08:19:02.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:02.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj43-p1 td/osd-recovery-prio/data 2024-04-25T08:19:03.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:03.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj44-p1 td/osd-recovery-prio/data 2024-04-25T08:19:04.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:04.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj45-p1 td/osd-recovery-prio/data 2024-04-25T08:19:05.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:05.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj46-p1 td/osd-recovery-prio/data 2024-04-25T08:19:06.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:06.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj47-p1 td/osd-recovery-prio/data 2024-04-25T08:19:07.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:07.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj48-p1 td/osd-recovery-prio/data 2024-04-25T08:19:08.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:08.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj49-p1 td/osd-recovery-prio/data 2024-04-25T08:19:09.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:09.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj50-p1 td/osd-recovery-prio/data 2024-04-25T08:19:10.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:10.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj51-p1 td/osd-recovery-prio/data 2024-04-25T08:19:11.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:11.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj52-p1 td/osd-recovery-prio/data 2024-04-25T08:19:12.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:12.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj53-p1 td/osd-recovery-prio/data 2024-04-25T08:19:13.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:13.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj54-p1 td/osd-recovery-prio/data 2024-04-25T08:19:14.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:14.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj55-p1 td/osd-recovery-prio/data 2024-04-25T08:19:15.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:15.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj56-p1 td/osd-recovery-prio/data 2024-04-25T08:19:16.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:16.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj57-p1 td/osd-recovery-prio/data 2024-04-25T08:19:17.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:17.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj58-p1 td/osd-recovery-prio/data 2024-04-25T08:19:18.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:18.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj59-p1 td/osd-recovery-prio/data 2024-04-25T08:19:19.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:19.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj60-p1 td/osd-recovery-prio/data 2024-04-25T08:19:21.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:21.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj61-p1 td/osd-recovery-prio/data 2024-04-25T08:19:21.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:21.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj62-p1 td/osd-recovery-prio/data 2024-04-25T08:19:22.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:22.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj63-p1 td/osd-recovery-prio/data 2024-04-25T08:19:23.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:23.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj64-p1 td/osd-recovery-prio/data 2024-04-25T08:19:24.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:24.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj65-p1 td/osd-recovery-prio/data 2024-04-25T08:19:25.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:25.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj66-p1 td/osd-recovery-prio/data 2024-04-25T08:19:26.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:26.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj67-p1 td/osd-recovery-prio/data 2024-04-25T08:19:27.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:27.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj68-p1 td/osd-recovery-prio/data 2024-04-25T08:19:28.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:28.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj69-p1 td/osd-recovery-prio/data 2024-04-25T08:19:29.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:29.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj70-p1 td/osd-recovery-prio/data 2024-04-25T08:19:30.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:30.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj71-p1 td/osd-recovery-prio/data 2024-04-25T08:19:31.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:31.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj72-p1 td/osd-recovery-prio/data 2024-04-25T08:19:32.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:32.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj73-p1 td/osd-recovery-prio/data 2024-04-25T08:19:33.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:33.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj74-p1 td/osd-recovery-prio/data 2024-04-25T08:19:34.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:34.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj75-p1 td/osd-recovery-prio/data 2024-04-25T08:19:35.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:35.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj76-p1 td/osd-recovery-prio/data 2024-04-25T08:19:36.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:36.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj77-p1 td/osd-recovery-prio/data 2024-04-25T08:19:37.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:37.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj78-p1 td/osd-recovery-prio/data 2024-04-25T08:19:38.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:38.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj79-p1 td/osd-recovery-prio/data 2024-04-25T08:19:39.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:39.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj80-p1 td/osd-recovery-prio/data 2024-04-25T08:19:40.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:40.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj81-p1 td/osd-recovery-prio/data 2024-04-25T08:19:41.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:41.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj82-p1 td/osd-recovery-prio/data 2024-04-25T08:19:42.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:42.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj83-p1 td/osd-recovery-prio/data 2024-04-25T08:19:43.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:43.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj84-p1 td/osd-recovery-prio/data 2024-04-25T08:19:44.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:44.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj85-p1 td/osd-recovery-prio/data 2024-04-25T08:19:46.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:46.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj86-p1 td/osd-recovery-prio/data 2024-04-25T08:19:47.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:47.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj87-p1 td/osd-recovery-prio/data 2024-04-25T08:19:48.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:48.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj88-p1 td/osd-recovery-prio/data 2024-04-25T08:19:48.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:48.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj89-p1 td/osd-recovery-prio/data 2024-04-25T08:19:49.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:49.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj90-p1 td/osd-recovery-prio/data 2024-04-25T08:19:50.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:50.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj91-p1 td/osd-recovery-prio/data 2024-04-25T08:19:51.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:51.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj92-p1 td/osd-recovery-prio/data 2024-04-25T08:19:52.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:52.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj93-p1 td/osd-recovery-prio/data 2024-04-25T08:19:53.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:53.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj94-p1 td/osd-recovery-prio/data 2024-04-25T08:19:54.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:54.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj95-p1 td/osd-recovery-prio/data 2024-04-25T08:19:55.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:55.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj96-p1 td/osd-recovery-prio/data 2024-04-25T08:19:56.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:56.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj97-p1 td/osd-recovery-prio/data 2024-04-25T08:19:57.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:57.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj98-p1 td/osd-recovery-prio/data 2024-04-25T08:19:58.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:58.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj99-p1 td/osd-recovery-prio/data 2024-04-25T08:19:59.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:19:59.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj100-p1 td/osd-recovery-prio/data 2024-04-25T08:20:01.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:01.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj101-p1 td/osd-recovery-prio/data 2024-04-25T08:20:02.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:02.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj102-p1 td/osd-recovery-prio/data 2024-04-25T08:20:03.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:03.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj103-p1 td/osd-recovery-prio/data 2024-04-25T08:20:04.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:04.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj104-p1 td/osd-recovery-prio/data 2024-04-25T08:20:05.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:05.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj105-p1 td/osd-recovery-prio/data 2024-04-25T08:20:06.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:06.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj106-p1 td/osd-recovery-prio/data 2024-04-25T08:20:07.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:07.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj107-p1 td/osd-recovery-prio/data 2024-04-25T08:20:08.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:08.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj108-p1 td/osd-recovery-prio/data 2024-04-25T08:20:09.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:09.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj109-p1 td/osd-recovery-prio/data 2024-04-25T08:20:10.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:10.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj110-p1 td/osd-recovery-prio/data 2024-04-25T08:20:11.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:11.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj111-p1 td/osd-recovery-prio/data 2024-04-25T08:20:12.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:12.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj112-p1 td/osd-recovery-prio/data 2024-04-25T08:20:13.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:13.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj113-p1 td/osd-recovery-prio/data 2024-04-25T08:20:14.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:14.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj114-p1 td/osd-recovery-prio/data 2024-04-25T08:20:14.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:14.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj115-p1 td/osd-recovery-prio/data 2024-04-25T08:20:15.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:15.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj116-p1 td/osd-recovery-prio/data 2024-04-25T08:20:16.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:16.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj117-p1 td/osd-recovery-prio/data 2024-04-25T08:20:17.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:17.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj118-p1 td/osd-recovery-prio/data 2024-04-25T08:20:18.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:18.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj119-p1 td/osd-recovery-prio/data 2024-04-25T08:20:19.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:19.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj120-p1 td/osd-recovery-prio/data 2024-04-25T08:20:21.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:21.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj121-p1 td/osd-recovery-prio/data 2024-04-25T08:20:22.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:22.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj122-p1 td/osd-recovery-prio/data 2024-04-25T08:20:23.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:23.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj123-p1 td/osd-recovery-prio/data 2024-04-25T08:20:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj124-p1 td/osd-recovery-prio/data 2024-04-25T08:20:24.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:24.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj125-p1 td/osd-recovery-prio/data 2024-04-25T08:20:26.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:26.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj126-p1 td/osd-recovery-prio/data 2024-04-25T08:20:27.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:27.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj127-p1 td/osd-recovery-prio/data 2024-04-25T08:20:28.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:28.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj128-p1 td/osd-recovery-prio/data 2024-04-25T08:20:28.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:28.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj129-p1 td/osd-recovery-prio/data 2024-04-25T08:20:29.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:29.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj130-p1 td/osd-recovery-prio/data 2024-04-25T08:20:31.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:31.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj131-p1 td/osd-recovery-prio/data 2024-04-25T08:20:32.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:32.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj132-p1 td/osd-recovery-prio/data 2024-04-25T08:20:33.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:33.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj133-p1 td/osd-recovery-prio/data 2024-04-25T08:20:34.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:34.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj134-p1 td/osd-recovery-prio/data 2024-04-25T08:20:34.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:34.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj135-p1 td/osd-recovery-prio/data 2024-04-25T08:20:36.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:36.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj136-p1 td/osd-recovery-prio/data 2024-04-25T08:20:37.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:37.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj137-p1 td/osd-recovery-prio/data 2024-04-25T08:20:38.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:38.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj138-p1 td/osd-recovery-prio/data 2024-04-25T08:20:39.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:39.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj139-p1 td/osd-recovery-prio/data 2024-04-25T08:20:40.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:40.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj140-p1 td/osd-recovery-prio/data 2024-04-25T08:20:41.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:41.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj141-p1 td/osd-recovery-prio/data 2024-04-25T08:20:42.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:42.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj142-p1 td/osd-recovery-prio/data 2024-04-25T08:20:43.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:43.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj143-p1 td/osd-recovery-prio/data 2024-04-25T08:20:44.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:44.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj144-p1 td/osd-recovery-prio/data 2024-04-25T08:20:45.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:45.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj145-p1 td/osd-recovery-prio/data 2024-04-25T08:20:45.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:45.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj146-p1 td/osd-recovery-prio/data 2024-04-25T08:20:46.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:46.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj147-p1 td/osd-recovery-prio/data 2024-04-25T08:20:47.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:47.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj148-p1 td/osd-recovery-prio/data 2024-04-25T08:20:48.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:48.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj149-p1 td/osd-recovery-prio/data 2024-04-25T08:20:49.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:49.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj150-p1 td/osd-recovery-prio/data 2024-04-25T08:20:50.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:50.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj151-p1 td/osd-recovery-prio/data 2024-04-25T08:20:51.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:51.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj152-p1 td/osd-recovery-prio/data 2024-04-25T08:20:52.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:52.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj153-p1 td/osd-recovery-prio/data 2024-04-25T08:20:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj154-p1 td/osd-recovery-prio/data 2024-04-25T08:20:54.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:54.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj155-p1 td/osd-recovery-prio/data 2024-04-25T08:20:55.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:55.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj156-p1 td/osd-recovery-prio/data 2024-04-25T08:20:56.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:56.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj157-p1 td/osd-recovery-prio/data 2024-04-25T08:20:57.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:57.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj158-p1 td/osd-recovery-prio/data 2024-04-25T08:20:58.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:58.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj159-p1 td/osd-recovery-prio/data 2024-04-25T08:20:59.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:20:59.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj160-p1 td/osd-recovery-prio/data 2024-04-25T08:21:00.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:00.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj161-p1 td/osd-recovery-prio/data 2024-04-25T08:21:01.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:01.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj162-p1 td/osd-recovery-prio/data 2024-04-25T08:21:02.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:02.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj163-p1 td/osd-recovery-prio/data 2024-04-25T08:21:04.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:04.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj164-p1 td/osd-recovery-prio/data 2024-04-25T08:21:05.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:05.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj165-p1 td/osd-recovery-prio/data 2024-04-25T08:21:05.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:05.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj166-p1 td/osd-recovery-prio/data 2024-04-25T08:21:07.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:07.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj167-p1 td/osd-recovery-prio/data 2024-04-25T08:21:08.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:08.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj168-p1 td/osd-recovery-prio/data 2024-04-25T08:21:09.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:09.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj169-p1 td/osd-recovery-prio/data 2024-04-25T08:21:10.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:10.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj170-p1 td/osd-recovery-prio/data 2024-04-25T08:21:11.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:11.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj171-p1 td/osd-recovery-prio/data 2024-04-25T08:21:12.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:12.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj172-p1 td/osd-recovery-prio/data 2024-04-25T08:21:13.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:13.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj173-p1 td/osd-recovery-prio/data 2024-04-25T08:21:14.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:14.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj174-p1 td/osd-recovery-prio/data 2024-04-25T08:21:15.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:15.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj175-p1 td/osd-recovery-prio/data 2024-04-25T08:21:16.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:16.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj176-p1 td/osd-recovery-prio/data 2024-04-25T08:21:17.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:17.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj177-p1 td/osd-recovery-prio/data 2024-04-25T08:21:17.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:17.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj178-p1 td/osd-recovery-prio/data 2024-04-25T08:21:19.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:19.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj179-p1 td/osd-recovery-prio/data 2024-04-25T08:21:19.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:19.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj180-p1 td/osd-recovery-prio/data 2024-04-25T08:21:21.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:21.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj181-p1 td/osd-recovery-prio/data 2024-04-25T08:21:22.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:22.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj182-p1 td/osd-recovery-prio/data 2024-04-25T08:21:23.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:23.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj183-p1 td/osd-recovery-prio/data 2024-04-25T08:21:24.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:24.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj184-p1 td/osd-recovery-prio/data 2024-04-25T08:21:25.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:25.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj185-p1 td/osd-recovery-prio/data 2024-04-25T08:21:26.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:26.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj186-p1 td/osd-recovery-prio/data 2024-04-25T08:21:27.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:27.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj187-p1 td/osd-recovery-prio/data 2024-04-25T08:21:28.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:28.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj188-p1 td/osd-recovery-prio/data 2024-04-25T08:21:28.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:28.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj189-p1 td/osd-recovery-prio/data 2024-04-25T08:21:30.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:30.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj190-p1 td/osd-recovery-prio/data 2024-04-25T08:21:30.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:30.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj191-p1 td/osd-recovery-prio/data 2024-04-25T08:21:32.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:32.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj192-p1 td/osd-recovery-prio/data 2024-04-25T08:21:33.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:33.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj193-p1 td/osd-recovery-prio/data 2024-04-25T08:21:34.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:34.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj194-p1 td/osd-recovery-prio/data 2024-04-25T08:21:35.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:35.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj195-p1 td/osd-recovery-prio/data 2024-04-25T08:21:36.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:36.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj196-p1 td/osd-recovery-prio/data 2024-04-25T08:21:37.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:37.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj197-p1 td/osd-recovery-prio/data 2024-04-25T08:21:38.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:38.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj198-p1 td/osd-recovery-prio/data 2024-04-25T08:21:39.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:39.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj199-p1 td/osd-recovery-prio/data 2024-04-25T08:21:40.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:40.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test1 put obj200-p1 td/osd-recovery-prio/data 2024-04-25T08:21:41.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 1 + 1 2024-04-25T08:21:41.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=2 2024-04-25T08:21:41.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T08:21:41.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T08:21:41.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:41.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj1-p2 td/osd-recovery-prio/data 2024-04-25T08:21:42.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:42.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj2-p2 td/osd-recovery-prio/data 2024-04-25T08:21:42.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:42.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj3-p2 td/osd-recovery-prio/data 2024-04-25T08:21:43.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:43.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj4-p2 td/osd-recovery-prio/data 2024-04-25T08:21:43.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:43.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj5-p2 td/osd-recovery-prio/data 2024-04-25T08:21:44.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:44.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj6-p2 td/osd-recovery-prio/data 2024-04-25T08:21:45.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:45.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj7-p2 td/osd-recovery-prio/data 2024-04-25T08:21:45.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:45.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj8-p2 td/osd-recovery-prio/data 2024-04-25T08:21:46.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:46.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj9-p2 td/osd-recovery-prio/data 2024-04-25T08:21:47.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:47.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj10-p2 td/osd-recovery-prio/data 2024-04-25T08:21:47.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:47.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj11-p2 td/osd-recovery-prio/data 2024-04-25T08:21:48.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:48.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj12-p2 td/osd-recovery-prio/data 2024-04-25T08:21:48.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:48.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj13-p2 td/osd-recovery-prio/data 2024-04-25T08:21:49.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:49.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj14-p2 td/osd-recovery-prio/data 2024-04-25T08:21:50.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:50.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj15-p2 td/osd-recovery-prio/data 2024-04-25T08:21:50.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:50.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj16-p2 td/osd-recovery-prio/data 2024-04-25T08:21:51.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:51.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj17-p2 td/osd-recovery-prio/data 2024-04-25T08:21:51.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:51.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj18-p2 td/osd-recovery-prio/data 2024-04-25T08:21:52.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:52.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj19-p2 td/osd-recovery-prio/data 2024-04-25T08:21:53.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:53.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj20-p2 td/osd-recovery-prio/data 2024-04-25T08:21:53.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:53.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj21-p2 td/osd-recovery-prio/data 2024-04-25T08:21:54.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:54.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj22-p2 td/osd-recovery-prio/data 2024-04-25T08:21:54.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:54.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj23-p2 td/osd-recovery-prio/data 2024-04-25T08:21:55.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:55.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj24-p2 td/osd-recovery-prio/data 2024-04-25T08:21:56.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:56.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj25-p2 td/osd-recovery-prio/data 2024-04-25T08:21:56.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:56.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj26-p2 td/osd-recovery-prio/data 2024-04-25T08:21:57.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:57.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj27-p2 td/osd-recovery-prio/data 2024-04-25T08:21:57.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:57.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj28-p2 td/osd-recovery-prio/data 2024-04-25T08:21:58.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:58.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj29-p2 td/osd-recovery-prio/data 2024-04-25T08:21:59.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:59.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj30-p2 td/osd-recovery-prio/data 2024-04-25T08:21:59.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:21:59.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj31-p2 td/osd-recovery-prio/data 2024-04-25T08:22:00.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:00.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj32-p2 td/osd-recovery-prio/data 2024-04-25T08:22:01.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:01.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj33-p2 td/osd-recovery-prio/data 2024-04-25T08:22:01.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:01.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj34-p2 td/osd-recovery-prio/data 2024-04-25T08:22:02.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:02.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj35-p2 td/osd-recovery-prio/data 2024-04-25T08:22:02.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:02.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj36-p2 td/osd-recovery-prio/data 2024-04-25T08:22:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:03.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj37-p2 td/osd-recovery-prio/data 2024-04-25T08:22:04.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:04.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj38-p2 td/osd-recovery-prio/data 2024-04-25T08:22:04.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:04.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj39-p2 td/osd-recovery-prio/data 2024-04-25T08:22:05.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:05.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj40-p2 td/osd-recovery-prio/data 2024-04-25T08:22:06.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:06.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj41-p2 td/osd-recovery-prio/data 2024-04-25T08:22:06.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:06.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj42-p2 td/osd-recovery-prio/data 2024-04-25T08:22:07.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:07.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj43-p2 td/osd-recovery-prio/data 2024-04-25T08:22:07.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:07.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj44-p2 td/osd-recovery-prio/data 2024-04-25T08:22:08.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:08.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj45-p2 td/osd-recovery-prio/data 2024-04-25T08:22:09.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:09.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj46-p2 td/osd-recovery-prio/data 2024-04-25T08:22:09.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:09.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj47-p2 td/osd-recovery-prio/data 2024-04-25T08:22:10.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:10.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj48-p2 td/osd-recovery-prio/data 2024-04-25T08:22:11.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:11.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj49-p2 td/osd-recovery-prio/data 2024-04-25T08:22:11.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:11.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj50-p2 td/osd-recovery-prio/data 2024-04-25T08:22:12.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:12.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj51-p2 td/osd-recovery-prio/data 2024-04-25T08:22:13.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:13.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj52-p2 td/osd-recovery-prio/data 2024-04-25T08:22:13.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:13.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj53-p2 td/osd-recovery-prio/data 2024-04-25T08:22:14.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:14.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj54-p2 td/osd-recovery-prio/data 2024-04-25T08:22:15.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:15.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj55-p2 td/osd-recovery-prio/data 2024-04-25T08:22:15.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:15.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj56-p2 td/osd-recovery-prio/data 2024-04-25T08:22:16.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:16.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj57-p2 td/osd-recovery-prio/data 2024-04-25T08:22:16.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:16.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj58-p2 td/osd-recovery-prio/data 2024-04-25T08:22:17.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:17.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj59-p2 td/osd-recovery-prio/data 2024-04-25T08:22:17.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:17.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj60-p2 td/osd-recovery-prio/data 2024-04-25T08:22:18.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:18.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj61-p2 td/osd-recovery-prio/data 2024-04-25T08:22:19.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:19.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj62-p2 td/osd-recovery-prio/data 2024-04-25T08:22:19.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:19.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj63-p2 td/osd-recovery-prio/data 2024-04-25T08:22:20.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:20.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj64-p2 td/osd-recovery-prio/data 2024-04-25T08:22:21.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:21.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj65-p2 td/osd-recovery-prio/data 2024-04-25T08:22:21.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:21.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj66-p2 td/osd-recovery-prio/data 2024-04-25T08:22:22.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:22.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj67-p2 td/osd-recovery-prio/data 2024-04-25T08:22:22.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:22.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj68-p2 td/osd-recovery-prio/data 2024-04-25T08:22:23.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:23.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj69-p2 td/osd-recovery-prio/data 2024-04-25T08:22:23.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:23.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj70-p2 td/osd-recovery-prio/data 2024-04-25T08:22:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:24.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj71-p2 td/osd-recovery-prio/data 2024-04-25T08:22:25.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:25.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj72-p2 td/osd-recovery-prio/data 2024-04-25T08:22:25.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:25.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj73-p2 td/osd-recovery-prio/data 2024-04-25T08:22:26.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:26.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj74-p2 td/osd-recovery-prio/data 2024-04-25T08:22:26.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:26.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj75-p2 td/osd-recovery-prio/data 2024-04-25T08:22:27.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:27.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj76-p2 td/osd-recovery-prio/data 2024-04-25T08:22:28.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:28.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj77-p2 td/osd-recovery-prio/data 2024-04-25T08:22:28.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:28.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj78-p2 td/osd-recovery-prio/data 2024-04-25T08:22:29.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:29.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj79-p2 td/osd-recovery-prio/data 2024-04-25T08:22:29.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:29.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj80-p2 td/osd-recovery-prio/data 2024-04-25T08:22:30.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:30.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj81-p2 td/osd-recovery-prio/data 2024-04-25T08:22:31.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:31.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj82-p2 td/osd-recovery-prio/data 2024-04-25T08:22:31.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:31.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj83-p2 td/osd-recovery-prio/data 2024-04-25T08:22:32.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:32.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj84-p2 td/osd-recovery-prio/data 2024-04-25T08:22:33.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:33.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj85-p2 td/osd-recovery-prio/data 2024-04-25T08:22:33.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:33.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj86-p2 td/osd-recovery-prio/data 2024-04-25T08:22:34.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:34.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj87-p2 td/osd-recovery-prio/data 2024-04-25T08:22:35.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:35.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj88-p2 td/osd-recovery-prio/data 2024-04-25T08:22:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:35.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj89-p2 td/osd-recovery-prio/data 2024-04-25T08:22:36.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:36.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj90-p2 td/osd-recovery-prio/data 2024-04-25T08:22:36.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:36.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj91-p2 td/osd-recovery-prio/data 2024-04-25T08:22:37.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:37.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj92-p2 td/osd-recovery-prio/data 2024-04-25T08:22:38.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:38.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj93-p2 td/osd-recovery-prio/data 2024-04-25T08:22:38.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:38.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj94-p2 td/osd-recovery-prio/data 2024-04-25T08:22:39.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:39.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj95-p2 td/osd-recovery-prio/data 2024-04-25T08:22:40.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:40.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj96-p2 td/osd-recovery-prio/data 2024-04-25T08:22:40.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:40.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj97-p2 td/osd-recovery-prio/data 2024-04-25T08:22:41.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:41.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj98-p2 td/osd-recovery-prio/data 2024-04-25T08:22:41.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:41.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj99-p2 td/osd-recovery-prio/data 2024-04-25T08:22:42.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:42.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj100-p2 td/osd-recovery-prio/data 2024-04-25T08:22:43.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:43.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj101-p2 td/osd-recovery-prio/data 2024-04-25T08:22:43.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:43.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj102-p2 td/osd-recovery-prio/data 2024-04-25T08:22:44.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:44.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj103-p2 td/osd-recovery-prio/data 2024-04-25T08:22:44.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:44.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj104-p2 td/osd-recovery-prio/data 2024-04-25T08:22:45.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:45.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj105-p2 td/osd-recovery-prio/data 2024-04-25T08:22:46.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:46.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj106-p2 td/osd-recovery-prio/data 2024-04-25T08:22:46.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:46.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj107-p2 td/osd-recovery-prio/data 2024-04-25T08:22:47.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:47.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj108-p2 td/osd-recovery-prio/data 2024-04-25T08:22:47.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:47.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj109-p2 td/osd-recovery-prio/data 2024-04-25T08:22:48.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:48.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj110-p2 td/osd-recovery-prio/data 2024-04-25T08:22:49.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:49.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj111-p2 td/osd-recovery-prio/data 2024-04-25T08:22:49.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:49.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj112-p2 td/osd-recovery-prio/data 2024-04-25T08:22:50.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:50.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj113-p2 td/osd-recovery-prio/data 2024-04-25T08:22:51.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:51.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj114-p2 td/osd-recovery-prio/data 2024-04-25T08:22:51.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:51.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj115-p2 td/osd-recovery-prio/data 2024-04-25T08:22:52.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:52.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj116-p2 td/osd-recovery-prio/data 2024-04-25T08:22:52.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:52.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj117-p2 td/osd-recovery-prio/data 2024-04-25T08:22:53.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:53.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj118-p2 td/osd-recovery-prio/data 2024-04-25T08:22:54.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:54.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj119-p2 td/osd-recovery-prio/data 2024-04-25T08:22:54.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:54.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj120-p2 td/osd-recovery-prio/data 2024-04-25T08:22:55.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:55.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj121-p2 td/osd-recovery-prio/data 2024-04-25T08:22:55.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:55.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj122-p2 td/osd-recovery-prio/data 2024-04-25T08:22:56.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:56.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj123-p2 td/osd-recovery-prio/data 2024-04-25T08:22:57.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:57.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj124-p2 td/osd-recovery-prio/data 2024-04-25T08:22:57.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:57.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj125-p2 td/osd-recovery-prio/data 2024-04-25T08:22:58.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:58.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj126-p2 td/osd-recovery-prio/data 2024-04-25T08:22:59.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:59.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj127-p2 td/osd-recovery-prio/data 2024-04-25T08:22:59.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:22:59.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj128-p2 td/osd-recovery-prio/data 2024-04-25T08:23:00.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:00.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj129-p2 td/osd-recovery-prio/data 2024-04-25T08:23:00.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:00.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj130-p2 td/osd-recovery-prio/data 2024-04-25T08:23:01.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:01.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj131-p2 td/osd-recovery-prio/data 2024-04-25T08:23:02.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:02.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj132-p2 td/osd-recovery-prio/data 2024-04-25T08:23:02.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:02.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj133-p2 td/osd-recovery-prio/data 2024-04-25T08:23:03.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:03.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj134-p2 td/osd-recovery-prio/data 2024-04-25T08:23:03.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:03.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj135-p2 td/osd-recovery-prio/data 2024-04-25T08:23:04.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:04.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj136-p2 td/osd-recovery-prio/data 2024-04-25T08:23:05.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:05.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj137-p2 td/osd-recovery-prio/data 2024-04-25T08:23:05.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:05.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj138-p2 td/osd-recovery-prio/data 2024-04-25T08:23:06.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:06.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj139-p2 td/osd-recovery-prio/data 2024-04-25T08:23:07.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:07.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj140-p2 td/osd-recovery-prio/data 2024-04-25T08:23:07.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:07.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj141-p2 td/osd-recovery-prio/data 2024-04-25T08:23:08.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:08.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj142-p2 td/osd-recovery-prio/data 2024-04-25T08:23:09.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:09.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj143-p2 td/osd-recovery-prio/data 2024-04-25T08:23:09.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:09.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj144-p2 td/osd-recovery-prio/data 2024-04-25T08:23:10.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:10.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj145-p2 td/osd-recovery-prio/data 2024-04-25T08:23:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:11.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj146-p2 td/osd-recovery-prio/data 2024-04-25T08:23:11.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:11.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj147-p2 td/osd-recovery-prio/data 2024-04-25T08:23:12.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:12.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj148-p2 td/osd-recovery-prio/data 2024-04-25T08:23:12.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:12.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj149-p2 td/osd-recovery-prio/data 2024-04-25T08:23:13.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:13.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj150-p2 td/osd-recovery-prio/data 2024-04-25T08:23:14.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:14.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj151-p2 td/osd-recovery-prio/data 2024-04-25T08:23:14.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:14.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj152-p2 td/osd-recovery-prio/data 2024-04-25T08:23:15.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:15.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj153-p2 td/osd-recovery-prio/data 2024-04-25T08:23:15.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:15.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj154-p2 td/osd-recovery-prio/data 2024-04-25T08:23:16.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:16.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj155-p2 td/osd-recovery-prio/data 2024-04-25T08:23:17.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:17.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj156-p2 td/osd-recovery-prio/data 2024-04-25T08:23:17.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:17.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj157-p2 td/osd-recovery-prio/data 2024-04-25T08:23:18.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:18.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj158-p2 td/osd-recovery-prio/data 2024-04-25T08:23:19.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:19.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj159-p2 td/osd-recovery-prio/data 2024-04-25T08:23:19.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:19.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj160-p2 td/osd-recovery-prio/data 2024-04-25T08:23:20.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:20.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj161-p2 td/osd-recovery-prio/data 2024-04-25T08:23:20.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:20.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj162-p2 td/osd-recovery-prio/data 2024-04-25T08:23:21.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:21.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj163-p2 td/osd-recovery-prio/data 2024-04-25T08:23:22.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:22.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj164-p2 td/osd-recovery-prio/data 2024-04-25T08:23:22.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:22.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj165-p2 td/osd-recovery-prio/data 2024-04-25T08:23:23.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:23.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj166-p2 td/osd-recovery-prio/data 2024-04-25T08:23:23.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:23.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj167-p2 td/osd-recovery-prio/data 2024-04-25T08:23:24.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:24.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj168-p2 td/osd-recovery-prio/data 2024-04-25T08:23:25.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:25.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj169-p2 td/osd-recovery-prio/data 2024-04-25T08:23:25.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:25.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj170-p2 td/osd-recovery-prio/data 2024-04-25T08:23:26.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:26.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj171-p2 td/osd-recovery-prio/data 2024-04-25T08:23:27.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:27.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj172-p2 td/osd-recovery-prio/data 2024-04-25T08:23:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj173-p2 td/osd-recovery-prio/data 2024-04-25T08:23:28.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:28.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj174-p2 td/osd-recovery-prio/data 2024-04-25T08:23:28.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:28.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj175-p2 td/osd-recovery-prio/data 2024-04-25T08:23:29.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:29.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj176-p2 td/osd-recovery-prio/data 2024-04-25T08:23:30.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:30.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj177-p2 td/osd-recovery-prio/data 2024-04-25T08:23:30.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:30.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj178-p2 td/osd-recovery-prio/data 2024-04-25T08:23:31.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:31.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj179-p2 td/osd-recovery-prio/data 2024-04-25T08:23:31.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:31.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj180-p2 td/osd-recovery-prio/data 2024-04-25T08:23:32.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:32.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj181-p2 td/osd-recovery-prio/data 2024-04-25T08:23:33.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:33.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj182-p2 td/osd-recovery-prio/data 2024-04-25T08:23:33.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:33.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj183-p2 td/osd-recovery-prio/data 2024-04-25T08:23:34.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:34.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj184-p2 td/osd-recovery-prio/data 2024-04-25T08:23:35.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:35.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj185-p2 td/osd-recovery-prio/data 2024-04-25T08:23:35.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:35.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj186-p2 td/osd-recovery-prio/data 2024-04-25T08:23:36.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:36.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj187-p2 td/osd-recovery-prio/data 2024-04-25T08:23:37.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:37.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj188-p2 td/osd-recovery-prio/data 2024-04-25T08:23:37.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:37.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj189-p2 td/osd-recovery-prio/data 2024-04-25T08:23:38.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:38.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj190-p2 td/osd-recovery-prio/data 2024-04-25T08:23:38.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:38.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj191-p2 td/osd-recovery-prio/data 2024-04-25T08:23:39.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:39.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj192-p2 td/osd-recovery-prio/data 2024-04-25T08:23:39.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:39.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj193-p2 td/osd-recovery-prio/data 2024-04-25T08:23:40.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:40.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj194-p2 td/osd-recovery-prio/data 2024-04-25T08:23:41.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:41.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj195-p2 td/osd-recovery-prio/data 2024-04-25T08:23:41.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:41.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj196-p2 td/osd-recovery-prio/data 2024-04-25T08:23:42.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:42.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj197-p2 td/osd-recovery-prio/data 2024-04-25T08:23:43.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:43.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj198-p2 td/osd-recovery-prio/data 2024-04-25T08:23:43.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:43.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj199-p2 td/osd-recovery-prio/data 2024-04-25T08:23:44.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:44.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test3 put obj200-p2 td/osd-recovery-prio/data 2024-04-25T08:23:45.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 2 + 1 2024-04-25T08:23:45.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=3 2024-04-25T08:23:45.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:137: TEST_recovery_priority: for pname in $pool1 $pool2 $pool3 2024-04-25T08:23:45.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: seq 1 200 2024-04-25T08:23:45.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:45.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj1-p3 td/osd-recovery-prio/data 2024-04-25T08:23:45.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:45.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj2-p3 td/osd-recovery-prio/data 2024-04-25T08:23:46.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:46.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj3-p3 td/osd-recovery-prio/data 2024-04-25T08:23:46.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:46.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj4-p3 td/osd-recovery-prio/data 2024-04-25T08:23:47.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:47.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj5-p3 td/osd-recovery-prio/data 2024-04-25T08:23:48.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:48.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj6-p3 td/osd-recovery-prio/data 2024-04-25T08:23:48.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:48.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj7-p3 td/osd-recovery-prio/data 2024-04-25T08:23:49.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:49.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj8-p3 td/osd-recovery-prio/data 2024-04-25T08:23:49.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:49.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj9-p3 td/osd-recovery-prio/data 2024-04-25T08:23:50.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:50.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj10-p3 td/osd-recovery-prio/data 2024-04-25T08:23:51.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:51.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj11-p3 td/osd-recovery-prio/data 2024-04-25T08:23:51.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:51.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj12-p3 td/osd-recovery-prio/data 2024-04-25T08:23:52.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:52.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj13-p3 td/osd-recovery-prio/data 2024-04-25T08:23:52.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:52.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj14-p3 td/osd-recovery-prio/data 2024-04-25T08:23:53.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:53.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj15-p3 td/osd-recovery-prio/data 2024-04-25T08:23:54.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:54.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj16-p3 td/osd-recovery-prio/data 2024-04-25T08:23:54.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:54.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj17-p3 td/osd-recovery-prio/data 2024-04-25T08:23:55.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:55.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj18-p3 td/osd-recovery-prio/data 2024-04-25T08:23:55.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:55.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj19-p3 td/osd-recovery-prio/data 2024-04-25T08:23:56.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:56.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj20-p3 td/osd-recovery-prio/data 2024-04-25T08:23:57.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:57.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj21-p3 td/osd-recovery-prio/data 2024-04-25T08:23:57.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:57.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj22-p3 td/osd-recovery-prio/data 2024-04-25T08:23:58.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:58.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj23-p3 td/osd-recovery-prio/data 2024-04-25T08:23:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:59.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj24-p3 td/osd-recovery-prio/data 2024-04-25T08:23:59.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:23:59.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj25-p3 td/osd-recovery-prio/data 2024-04-25T08:24:00.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:00.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj26-p3 td/osd-recovery-prio/data 2024-04-25T08:24:00.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:00.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj27-p3 td/osd-recovery-prio/data 2024-04-25T08:24:01.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:01.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj28-p3 td/osd-recovery-prio/data 2024-04-25T08:24:01.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:01.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj29-p3 td/osd-recovery-prio/data 2024-04-25T08:24:02.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:02.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj30-p3 td/osd-recovery-prio/data 2024-04-25T08:24:03.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:03.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj31-p3 td/osd-recovery-prio/data 2024-04-25T08:24:03.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:03.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj32-p3 td/osd-recovery-prio/data 2024-04-25T08:24:04.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:04.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj33-p3 td/osd-recovery-prio/data 2024-04-25T08:24:05.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:05.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj34-p3 td/osd-recovery-prio/data 2024-04-25T08:24:05.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:05.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj35-p3 td/osd-recovery-prio/data 2024-04-25T08:24:06.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:06.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj36-p3 td/osd-recovery-prio/data 2024-04-25T08:24:07.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:07.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj37-p3 td/osd-recovery-prio/data 2024-04-25T08:24:08.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:08.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj38-p3 td/osd-recovery-prio/data 2024-04-25T08:24:08.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:08.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj39-p3 td/osd-recovery-prio/data 2024-04-25T08:24:09.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:09.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj40-p3 td/osd-recovery-prio/data 2024-04-25T08:24:09.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:09.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj41-p3 td/osd-recovery-prio/data 2024-04-25T08:24:10.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:10.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj42-p3 td/osd-recovery-prio/data 2024-04-25T08:24:11.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:11.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj43-p3 td/osd-recovery-prio/data 2024-04-25T08:24:11.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:11.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj44-p3 td/osd-recovery-prio/data 2024-04-25T08:24:12.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:12.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj45-p3 td/osd-recovery-prio/data 2024-04-25T08:24:13.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:13.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj46-p3 td/osd-recovery-prio/data 2024-04-25T08:24:13.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:13.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj47-p3 td/osd-recovery-prio/data 2024-04-25T08:24:14.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:14.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj48-p3 td/osd-recovery-prio/data 2024-04-25T08:24:15.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:15.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj49-p3 td/osd-recovery-prio/data 2024-04-25T08:24:15.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:15.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj50-p3 td/osd-recovery-prio/data 2024-04-25T08:24:16.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:16.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj51-p3 td/osd-recovery-prio/data 2024-04-25T08:24:16.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:16.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj52-p3 td/osd-recovery-prio/data 2024-04-25T08:24:17.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:17.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj53-p3 td/osd-recovery-prio/data 2024-04-25T08:24:18.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:18.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj54-p3 td/osd-recovery-prio/data 2024-04-25T08:24:18.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:18.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj55-p3 td/osd-recovery-prio/data 2024-04-25T08:24:19.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:19.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj56-p3 td/osd-recovery-prio/data 2024-04-25T08:24:19.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:19.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj57-p3 td/osd-recovery-prio/data 2024-04-25T08:24:20.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:20.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj58-p3 td/osd-recovery-prio/data 2024-04-25T08:24:21.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:21.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj59-p3 td/osd-recovery-prio/data 2024-04-25T08:24:21.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:21.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj60-p3 td/osd-recovery-prio/data 2024-04-25T08:24:22.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:22.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj61-p3 td/osd-recovery-prio/data 2024-04-25T08:24:23.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:23.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj62-p3 td/osd-recovery-prio/data 2024-04-25T08:24:23.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:23.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj63-p3 td/osd-recovery-prio/data 2024-04-25T08:24:24.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:24.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj64-p3 td/osd-recovery-prio/data 2024-04-25T08:24:24.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:24.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj65-p3 td/osd-recovery-prio/data 2024-04-25T08:24:25.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:25.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj66-p3 td/osd-recovery-prio/data 2024-04-25T08:24:26.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:26.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj67-p3 td/osd-recovery-prio/data 2024-04-25T08:24:26.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:26.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj68-p3 td/osd-recovery-prio/data 2024-04-25T08:24:27.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:27.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj69-p3 td/osd-recovery-prio/data 2024-04-25T08:24:27.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:27.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj70-p3 td/osd-recovery-prio/data 2024-04-25T08:24:28.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:28.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj71-p3 td/osd-recovery-prio/data 2024-04-25T08:24:29.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:29.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj72-p3 td/osd-recovery-prio/data 2024-04-25T08:24:29.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:29.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj73-p3 td/osd-recovery-prio/data 2024-04-25T08:24:30.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:30.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj74-p3 td/osd-recovery-prio/data 2024-04-25T08:24:31.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:31.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj75-p3 td/osd-recovery-prio/data 2024-04-25T08:24:31.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:31.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj76-p3 td/osd-recovery-prio/data 2024-04-25T08:24:32.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:32.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj77-p3 td/osd-recovery-prio/data 2024-04-25T08:24:32.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:32.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj78-p3 td/osd-recovery-prio/data 2024-04-25T08:24:33.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:33.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj79-p3 td/osd-recovery-prio/data 2024-04-25T08:24:34.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:34.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj80-p3 td/osd-recovery-prio/data 2024-04-25T08:24:34.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:34.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj81-p3 td/osd-recovery-prio/data 2024-04-25T08:24:35.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:35.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj82-p3 td/osd-recovery-prio/data 2024-04-25T08:24:35.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:35.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj83-p3 td/osd-recovery-prio/data 2024-04-25T08:24:36.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:36.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj84-p3 td/osd-recovery-prio/data 2024-04-25T08:24:37.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:37.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj85-p3 td/osd-recovery-prio/data 2024-04-25T08:24:37.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:37.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj86-p3 td/osd-recovery-prio/data 2024-04-25T08:24:38.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:38.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj87-p3 td/osd-recovery-prio/data 2024-04-25T08:24:39.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:39.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj88-p3 td/osd-recovery-prio/data 2024-04-25T08:24:39.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:39.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj89-p3 td/osd-recovery-prio/data 2024-04-25T08:24:40.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:40.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj90-p3 td/osd-recovery-prio/data 2024-04-25T08:24:40.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:40.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj91-p3 td/osd-recovery-prio/data 2024-04-25T08:24:41.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:41.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj92-p3 td/osd-recovery-prio/data 2024-04-25T08:24:42.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:42.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj93-p3 td/osd-recovery-prio/data 2024-04-25T08:24:42.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:42.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj94-p3 td/osd-recovery-prio/data 2024-04-25T08:24:43.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:43.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj95-p3 td/osd-recovery-prio/data 2024-04-25T08:24:43.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:43.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj96-p3 td/osd-recovery-prio/data 2024-04-25T08:24:44.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:44.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj97-p3 td/osd-recovery-prio/data 2024-04-25T08:24:45.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:45.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj98-p3 td/osd-recovery-prio/data 2024-04-25T08:24:45.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:45.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj99-p3 td/osd-recovery-prio/data 2024-04-25T08:24:46.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:46.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj100-p3 td/osd-recovery-prio/data 2024-04-25T08:24:47.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:47.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj101-p3 td/osd-recovery-prio/data 2024-04-25T08:24:47.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:47.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj102-p3 td/osd-recovery-prio/data 2024-04-25T08:24:48.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:48.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj103-p3 td/osd-recovery-prio/data 2024-04-25T08:24:49.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:49.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj104-p3 td/osd-recovery-prio/data 2024-04-25T08:24:49.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:49.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj105-p3 td/osd-recovery-prio/data 2024-04-25T08:24:50.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:50.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj106-p3 td/osd-recovery-prio/data 2024-04-25T08:24:50.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:50.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj107-p3 td/osd-recovery-prio/data 2024-04-25T08:24:51.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:51.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj108-p3 td/osd-recovery-prio/data 2024-04-25T08:24:52.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:52.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj109-p3 td/osd-recovery-prio/data 2024-04-25T08:24:52.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:52.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj110-p3 td/osd-recovery-prio/data 2024-04-25T08:24:53.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:53.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj111-p3 td/osd-recovery-prio/data 2024-04-25T08:24:53.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:53.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj112-p3 td/osd-recovery-prio/data 2024-04-25T08:24:54.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:54.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj113-p3 td/osd-recovery-prio/data 2024-04-25T08:24:55.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:55.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj114-p3 td/osd-recovery-prio/data 2024-04-25T08:24:55.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:55.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj115-p3 td/osd-recovery-prio/data 2024-04-25T08:24:56.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:56.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj116-p3 td/osd-recovery-prio/data 2024-04-25T08:24:56.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:56.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj117-p3 td/osd-recovery-prio/data 2024-04-25T08:24:57.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:57.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj118-p3 td/osd-recovery-prio/data 2024-04-25T08:24:58.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:58.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj119-p3 td/osd-recovery-prio/data 2024-04-25T08:24:58.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:58.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj120-p3 td/osd-recovery-prio/data 2024-04-25T08:24:59.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:59.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj121-p3 td/osd-recovery-prio/data 2024-04-25T08:24:59.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:24:59.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj122-p3 td/osd-recovery-prio/data 2024-04-25T08:25:00.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:00.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj123-p3 td/osd-recovery-prio/data 2024-04-25T08:25:01.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:01.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj124-p3 td/osd-recovery-prio/data 2024-04-25T08:25:01.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:01.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj125-p3 td/osd-recovery-prio/data 2024-04-25T08:25:02.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:02.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj126-p3 td/osd-recovery-prio/data 2024-04-25T08:25:02.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:02.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj127-p3 td/osd-recovery-prio/data 2024-04-25T08:25:03.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:03.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj128-p3 td/osd-recovery-prio/data 2024-04-25T08:25:04.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:04.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj129-p3 td/osd-recovery-prio/data 2024-04-25T08:25:04.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:04.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj130-p3 td/osd-recovery-prio/data 2024-04-25T08:25:05.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:05.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj131-p3 td/osd-recovery-prio/data 2024-04-25T08:25:05.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:05.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj132-p3 td/osd-recovery-prio/data 2024-04-25T08:25:06.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:06.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj133-p3 td/osd-recovery-prio/data 2024-04-25T08:25:07.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:07.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj134-p3 td/osd-recovery-prio/data 2024-04-25T08:25:07.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:07.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj135-p3 td/osd-recovery-prio/data 2024-04-25T08:25:08.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:08.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj136-p3 td/osd-recovery-prio/data 2024-04-25T08:25:09.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:09.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj137-p3 td/osd-recovery-prio/data 2024-04-25T08:25:09.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:09.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj138-p3 td/osd-recovery-prio/data 2024-04-25T08:25:10.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:10.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj139-p3 td/osd-recovery-prio/data 2024-04-25T08:25:11.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:11.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj140-p3 td/osd-recovery-prio/data 2024-04-25T08:25:11.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:11.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj141-p3 td/osd-recovery-prio/data 2024-04-25T08:25:12.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:12.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj142-p3 td/osd-recovery-prio/data 2024-04-25T08:25:12.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:12.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj143-p3 td/osd-recovery-prio/data 2024-04-25T08:25:13.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:13.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj144-p3 td/osd-recovery-prio/data 2024-04-25T08:25:14.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:14.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj145-p3 td/osd-recovery-prio/data 2024-04-25T08:25:15.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:15.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj146-p3 td/osd-recovery-prio/data 2024-04-25T08:25:15.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:15.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj147-p3 td/osd-recovery-prio/data 2024-04-25T08:25:16.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:16.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj148-p3 td/osd-recovery-prio/data 2024-04-25T08:25:16.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:16.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj149-p3 td/osd-recovery-prio/data 2024-04-25T08:25:17.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:17.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj150-p3 td/osd-recovery-prio/data 2024-04-25T08:25:18.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:18.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj151-p3 td/osd-recovery-prio/data 2024-04-25T08:25:18.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:18.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj152-p3 td/osd-recovery-prio/data 2024-04-25T08:25:19.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:19.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj153-p3 td/osd-recovery-prio/data 2024-04-25T08:25:19.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:19.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj154-p3 td/osd-recovery-prio/data 2024-04-25T08:25:20.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:20.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj155-p3 td/osd-recovery-prio/data 2024-04-25T08:25:21.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:21.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj156-p3 td/osd-recovery-prio/data 2024-04-25T08:25:21.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:21.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj157-p3 td/osd-recovery-prio/data 2024-04-25T08:25:22.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:22.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj158-p3 td/osd-recovery-prio/data 2024-04-25T08:25:22.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:22.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj159-p3 td/osd-recovery-prio/data 2024-04-25T08:25:23.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:23.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj160-p3 td/osd-recovery-prio/data 2024-04-25T08:25:24.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:24.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj161-p3 td/osd-recovery-prio/data 2024-04-25T08:25:24.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:24.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj162-p3 td/osd-recovery-prio/data 2024-04-25T08:25:25.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:25.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj163-p3 td/osd-recovery-prio/data 2024-04-25T08:25:25.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:25.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj164-p3 td/osd-recovery-prio/data 2024-04-25T08:25:26.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:26.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj165-p3 td/osd-recovery-prio/data 2024-04-25T08:25:27.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:27.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj166-p3 td/osd-recovery-prio/data 2024-04-25T08:25:27.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:27.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj167-p3 td/osd-recovery-prio/data 2024-04-25T08:25:28.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:28.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj168-p3 td/osd-recovery-prio/data 2024-04-25T08:25:28.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:28.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj169-p3 td/osd-recovery-prio/data 2024-04-25T08:25:29.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:29.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj170-p3 td/osd-recovery-prio/data 2024-04-25T08:25:30.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:30.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj171-p3 td/osd-recovery-prio/data 2024-04-25T08:25:30.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:30.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj172-p3 td/osd-recovery-prio/data 2024-04-25T08:25:31.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:31.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj173-p3 td/osd-recovery-prio/data 2024-04-25T08:25:31.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:31.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj174-p3 td/osd-recovery-prio/data 2024-04-25T08:25:32.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:32.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj175-p3 td/osd-recovery-prio/data 2024-04-25T08:25:33.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:33.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj176-p3 td/osd-recovery-prio/data 2024-04-25T08:25:33.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:33.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj177-p3 td/osd-recovery-prio/data 2024-04-25T08:25:34.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:34.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj178-p3 td/osd-recovery-prio/data 2024-04-25T08:25:35.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:35.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj179-p3 td/osd-recovery-prio/data 2024-04-25T08:25:35.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:35.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj180-p3 td/osd-recovery-prio/data 2024-04-25T08:25:36.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:36.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj181-p3 td/osd-recovery-prio/data 2024-04-25T08:25:36.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:36.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj182-p3 td/osd-recovery-prio/data 2024-04-25T08:25:37.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:37.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj183-p3 td/osd-recovery-prio/data 2024-04-25T08:25:38.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:38.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj184-p3 td/osd-recovery-prio/data 2024-04-25T08:25:39.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:39.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj185-p3 td/osd-recovery-prio/data 2024-04-25T08:25:39.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:39.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj186-p3 td/osd-recovery-prio/data 2024-04-25T08:25:40.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:40.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj187-p3 td/osd-recovery-prio/data 2024-04-25T08:25:40.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:40.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj188-p3 td/osd-recovery-prio/data 2024-04-25T08:25:41.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:41.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj189-p3 td/osd-recovery-prio/data 2024-04-25T08:25:42.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:42.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj190-p3 td/osd-recovery-prio/data 2024-04-25T08:25:42.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:42.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj191-p3 td/osd-recovery-prio/data 2024-04-25T08:25:43.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:43.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj192-p3 td/osd-recovery-prio/data 2024-04-25T08:25:44.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:44.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj193-p3 td/osd-recovery-prio/data 2024-04-25T08:25:44.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:44.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj194-p3 td/osd-recovery-prio/data 2024-04-25T08:25:45.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:45.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj195-p3 td/osd-recovery-prio/data 2024-04-25T08:25:45.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:45.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj196-p3 td/osd-recovery-prio/data 2024-04-25T08:25:46.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:46.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj197-p3 td/osd-recovery-prio/data 2024-04-25T08:25:47.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:47.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj198-p3 td/osd-recovery-prio/data 2024-04-25T08:25:47.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:47.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj199-p3 td/osd-recovery-prio/data 2024-04-25T08:25:48.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:139: TEST_recovery_priority: for i in $(seq 1 $objects) 2024-04-25T08:25:48.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:141: TEST_recovery_priority: rados -p test7 put obj200-p3 td/osd-recovery-prio/data 2024-04-25T08:25:48.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: expr 3 + 1 2024-04-25T08:25:48.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:143: TEST_recovery_priority: p=4 2024-04-25T08:25:48.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: get_not_primary test1 obj1-p1 2024-04-25T08:25:48.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test1 2024-04-25T08:25:48.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1-p1 2024-04-25T08:25:48.993 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test1 obj1-p1 2024-04-25T08:25:48.993 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test1 2024-04-25T08:25:48.993 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1-p1 2024-04-25T08:25:48.993 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:25:48.994 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:25:49.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:25:49.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test1 obj1-p1 2024-04-25T08:25:49.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:25:49.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:146: TEST_recovery_priority: local otherosd=0 2024-04-25T08:25:49.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:148: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:25:50.370 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:25:50.370 INFO:tasks.workunit.client.0.smithi134.stdout:7.0 198 0 0 0 0 2069889024 0 0 592 0 592 active+clean 2024-04-25T08:18:13.977495+0000 87'592 87:664 [1] 1 [1] 1 0'0 2024-04-25T08:17:37.191617+0000 0'0 2024-04-25T08:17:37.191617+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:52:22.424586+0000 0 0 2024-04-25T08:25:50.370 INFO:tasks.workunit.client.0.smithi134.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:18:14.098793+0000 87'600 87:709 [1] 1 [1] 1 0'0 2024-04-25T08:17:23.969579+0000 0'0 2024-04-25T08:17:23.969579+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:24:47.087457+0000 0 0 2024-04-25T08:25:50.370 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:17:21.386899+0000 87'600 87:726 [1,0] 1 [1,0] 1 0'0 2024-04-25T08:17:17.811742+0000 0'0 2024-04-25T08:17:17.811742+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:20:36.850081+0000 0 0 2024-04-25T08:25:50.370 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:25:50.370 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:25:50.371 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:25:50.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:149: TEST_recovery_priority: ERRORS=0 2024-04-25T08:25:50.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:151: TEST_recovery_priority: ceph osd set norecover 2024-04-25T08:25:51.290 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is set 2024-04-25T08:25:51.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:152: TEST_recovery_priority: ceph osd set noout 2024-04-25T08:25:52.168 INFO:tasks.workunit.client.0.smithi134.stderr:noout is set 2024-04-25T08:25:52.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:156: TEST_recovery_priority: ceph osd pool set test7 size 2 2024-04-25T08:25:53.247 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 7 size to 2 2024-04-25T08:25:53.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:157: TEST_recovery_priority: sleep 2 2024-04-25T08:25:55.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:25:55.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:25:55.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:25:55.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:25:55.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:25:55.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:25:55.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:25:55.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:25:55.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:158: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:25:55.393 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:25:55.394 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:25:55.394 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:25:55.394 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:25:55.407 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: seq 1 10 2024-04-25T08:25:55.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:25:55.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:25:55.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:25:55.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:166: TEST_recovery_priority: '[' 1 = 10 ']' 2024-04-25T08:25:55.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:170: TEST_recovery_priority: sleep 2 2024-04-25T08:25:57.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:161: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:25:57.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:25:57.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:163: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:25:58.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:164: TEST_recovery_priority: break 2024-04-25T08:25:58.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:172: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:25:58.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:25:58.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:25:58.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:25:58.693 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:25:58.693 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:25:58.693 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:25:58.693 INFO:tasks.workunit.client.0.smithi134.stderr:4' 2024-04-25T08:25:58.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:25:58.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:25:58.693 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:25:58.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836603 2024-04-25T08:25:58.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836603 2024-04-25T08:25:58.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603' 2024-04-25T08:25:58.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:25:58.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:25:59.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673080 2024-04-25T08:25:59.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673080 2024-04-25T08:25:59.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673080' 2024-04-25T08:25:59.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:25:59.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:25:59.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509557 2024-04-25T08:25:59.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509557 2024-04-25T08:25:59.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673080 2-64424509557' 2024-04-25T08:25:59.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:25:59.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:25:59.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378738 2024-04-25T08:25:59.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378738 2024-04-25T08:25:59.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673080 2-64424509557 3-81604378738' 2024-04-25T08:25:59.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:25:59.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:25:59.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215215 2024-04-25T08:25:59.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215215 2024-04-25T08:25:59.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836603 1-42949673080 2-64424509557 3-81604378738 4-103079215215' 2024-04-25T08:25:59.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:25:59.460 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836603 2024-04-25T08:25:59.460 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:25:59.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:25:59.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836603 2024-04-25T08:25:59.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:25:59.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836603 2024-04-25T08:25:59.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836603' 2024-04-25T08:25:59.463 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836603 2024-04-25T08:25:59.463 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:25:59.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836601 -lt 21474836603 2024-04-25T08:25:59.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:26:00.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:26:00.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:01.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836603 -lt 21474836603 2024-04-25T08:26:01.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:01.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673080 2024-04-25T08:26:01.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:01.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:26:01.283 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673080 2024-04-25T08:26:01.283 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:01.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673080 2024-04-25T08:26:01.285 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673080 2024-04-25T08:26:01.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673080' 2024-04-25T08:26:01.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:26:01.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673080 -lt 42949673080 2024-04-25T08:26:01.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:01.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509557 2024-04-25T08:26:01.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:01.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:26:01.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509557 2024-04-25T08:26:01.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:01.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509557 2024-04-25T08:26:01.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509557' 2024-04-25T08:26:01.795 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509557 2024-04-25T08:26:01.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:26:02.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509557 -lt 64424509557 2024-04-25T08:26:02.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:02.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378738 2024-04-25T08:26:02.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:02.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:26:02.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378738 2024-04-25T08:26:02.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:02.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378738 2024-04-25T08:26:02.242 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378738 2024-04-25T08:26:02.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378738' 2024-04-25T08:26:02.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:26:02.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378738 -lt 81604378738 2024-04-25T08:26:02.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:02.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215215 2024-04-25T08:26:02.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:02.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:26:02.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215215 2024-04-25T08:26:02.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:02.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215215 2024-04-25T08:26:02.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215215' 2024-04-25T08:26:02.694 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215215 2024-04-25T08:26:02.694 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:26:03.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215215 -lt 103079215215 2024-04-25T08:26:03.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:26:03.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:26:03.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:26:03.119 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:26:03.120 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:03.120 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:26:03.120 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:26:03.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:26:03.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:173: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:03.228 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:03.229 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:03.229 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:03.229 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:03.229 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:03.229 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:03.229 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:03.229 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:03.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:175: TEST_recovery_priority: ceph osd out osd.0 2024-04-25T08:26:03.802 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T08:26:03.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:176: TEST_recovery_priority: sleep 2 2024-04-25T08:26:05.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:177: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:26:05.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:26:05.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:26:06.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:26:06.289 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:26:06.289 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:26:06.289 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:26:06.289 INFO:tasks.workunit.client.0.smithi134.stderr:4' 2024-04-25T08:26:06.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:26:06.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:06.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:26:06.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836606 2024-04-25T08:26:06.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836606 2024-04-25T08:26:06.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606' 2024-04-25T08:26:06.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:06.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:26:06.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673083 2024-04-25T08:26:06.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673083 2024-04-25T08:26:06.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673083' 2024-04-25T08:26:06.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:06.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:26:06.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509560 2024-04-25T08:26:06.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509560 2024-04-25T08:26:06.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673083 2-64424509560' 2024-04-25T08:26:06.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:06.770 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:26:06.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378741 2024-04-25T08:26:06.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378741 2024-04-25T08:26:06.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673083 2-64424509560 3-81604378741' 2024-04-25T08:26:06.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:06.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:26:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215218 2024-04-25T08:26:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215218 2024-04-25T08:26:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836606 1-42949673083 2-64424509560 3-81604378741 4-103079215218' 2024-04-25T08:26:07.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:07.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:07.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836606 2024-04-25T08:26:07.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:26:07.071 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836606 2024-04-25T08:26:07.071 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:07.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836606 2024-04-25T08:26:07.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836606' 2024-04-25T08:26:07.072 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836606 2024-04-25T08:26:07.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:07.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 0 -lt 21474836606 2024-04-25T08:26:07.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:26:08.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:26:08.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:08.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836607 -lt 21474836606 2024-04-25T08:26:08.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:08.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673083 2024-04-25T08:26:08.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:08.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:26:08.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673083 2024-04-25T08:26:08.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:08.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673083 2024-04-25T08:26:08.906 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673083 2024-04-25T08:26:08.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673083' 2024-04-25T08:26:08.906 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:26:09.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673084 -lt 42949673083 2024-04-25T08:26:09.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:09.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509560 2024-04-25T08:26:09.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:09.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:26:09.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509560 2024-04-25T08:26:09.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:09.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509560 2024-04-25T08:26:09.353 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509560 2024-04-25T08:26:09.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509560' 2024-04-25T08:26:09.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:26:09.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509561 -lt 64424509560 2024-04-25T08:26:09.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:09.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:09.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378741 2024-04-25T08:26:09.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:26:09.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378741 2024-04-25T08:26:09.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:09.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378741 2024-04-25T08:26:09.930 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378741 2024-04-25T08:26:09.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378741' 2024-04-25T08:26:09.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:26:10.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378742 -lt 81604378741 2024-04-25T08:26:10.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:10.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215218 2024-04-25T08:26:10.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:10.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:26:10.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215218 2024-04-25T08:26:10.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:10.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215218 2024-04-25T08:26:10.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215218' 2024-04-25T08:26:10.336 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215218 2024-04-25T08:26:10.336 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:26:10.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215219 -lt 103079215218 2024-04-25T08:26:10.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:26:10.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:26:10.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:26:10.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:26:10.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:10.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:26:10.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:26:10.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:26:10.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:178: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:10.940 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:10.940 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:10.940 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:10.940 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:10.941 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:10.942 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:10.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:179: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:26:11.370 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:26:11.371 INFO:tasks.workunit.client.0.smithi134.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-04-25T08:25:58.110944+0000 87'600 99:697 [1,4] 1 [1] 1 0'0 2024-04-25T08:17:37.191617+0000 0'0 2024-04-25T08:17:37.191617+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:26:11.371 INFO:tasks.workunit.client.0.smithi134.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:18:14.098793+0000 87'600 99:731 [1] 1 [1] 1 0'0 2024-04-25T08:17:23.969579+0000 0'0 2024-04-25T08:17:23.969579+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:24:47.087457+0000 0 0 2024-04-25T08:26:11.371 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T08:26:06.228296+0000 87'600 99:749 [1,2] 1 [1] 1 0'0 2024-04-25T08:17:17.811742+0000 0'0 2024-04-25T08:17:17.811742+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:26:11.371 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:26:11.371 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:26:11.371 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:26:11.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:181: TEST_recovery_priority: ceph osd pool set test3 size 2 2024-04-25T08:26:12.108 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 3 size to 2 2024-04-25T08:26:12.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:182: TEST_recovery_priority: sleep 2 2024-04-25T08:26:14.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:183: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:26:14.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:26:14.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:26:14.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:26:14.670 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:26:14.670 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:26:14.670 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:26:14.670 INFO:tasks.workunit.client.0.smithi134.stderr:4' 2024-04-25T08:26:14.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:26:14.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:14.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:26:14.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836610 2024-04-25T08:26:14.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836610 2024-04-25T08:26:14.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836610' 2024-04-25T08:26:14.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:14.937 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:26:15.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673087 2024-04-25T08:26:15.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673087 2024-04-25T08:26:15.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836610 1-42949673087' 2024-04-25T08:26:15.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:15.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:26:15.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509564 2024-04-25T08:26:15.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509564 2024-04-25T08:26:15.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836610 1-42949673087 2-64424509564' 2024-04-25T08:26:15.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:15.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:26:15.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378745 2024-04-25T08:26:15.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378745 2024-04-25T08:26:15.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836610 1-42949673087 2-64424509564 3-81604378745' 2024-04-25T08:26:15.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:15.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:26:15.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215222 2024-04-25T08:26:15.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215222 2024-04-25T08:26:15.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836610 1-42949673087 2-64424509564 3-81604378745 4-103079215222' 2024-04-25T08:26:15.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:15.562 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836610 2024-04-25T08:26:15.562 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:15.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:26:15.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836610 2024-04-25T08:26:15.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:15.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836610 2024-04-25T08:26:15.565 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836610 2024-04-25T08:26:15.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836610' 2024-04-25T08:26:15.565 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:15.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836608 -lt 21474836610 2024-04-25T08:26:15.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:26:16.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:26:16.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:17.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836610 -lt 21474836610 2024-04-25T08:26:17.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:17.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673087 2024-04-25T08:26:17.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:17.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:26:17.407 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673087 2024-04-25T08:26:17.407 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:17.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673087 2024-04-25T08:26:17.408 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673087 2024-04-25T08:26:17.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673087' 2024-04-25T08:26:17.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:26:17.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673087 -lt 42949673087 2024-04-25T08:26:17.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:17.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509564 2024-04-25T08:26:17.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:17.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:26:17.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509564 2024-04-25T08:26:17.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:17.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509564 2024-04-25T08:26:17.893 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509564 2024-04-25T08:26:17.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509564' 2024-04-25T08:26:17.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:26:18.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509565 -lt 64424509564 2024-04-25T08:26:18.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:18.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:18.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378745 2024-04-25T08:26:18.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:26:18.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378745 2024-04-25T08:26:18.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:18.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378745 2024-04-25T08:26:18.392 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378745 2024-04-25T08:26:18.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378745' 2024-04-25T08:26:18.393 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:26:18.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378746 -lt 81604378745 2024-04-25T08:26:18.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:18.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:18.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215222 2024-04-25T08:26:18.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:26:18.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215222 2024-04-25T08:26:18.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:18.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215222 2024-04-25T08:26:18.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215222' 2024-04-25T08:26:18.821 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215222 2024-04-25T08:26:18.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:26:19.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215223 -lt 103079215222 2024-04-25T08:26:19.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:26:19.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:26:19.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:26:19.234 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:26:19.235 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:19.235 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:26:19.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:26:19.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:26:19.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:184: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:19.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:185: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:19.344 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:19.344 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:26:19.345 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:19.346 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:19.347 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:19.347 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:19.347 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:19.347 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:19.347 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:19.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:186: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:26:19.732 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:26:19.732 INFO:tasks.workunit.client.0.smithi134.stdout:7.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovering+forced_recovery+undersized+degraded+remapped 2024-04-25T08:25:58.110944+0000 87'600 104:706 [1,4] 1 [1] 1 0'0 2024-04-25T08:17:37.191617+0000 0'0 2024-04-25T08:17:37.191617+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:26:19.732 INFO:tasks.workunit.client.0.smithi134.stdout:3.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T08:26:14.730667+0000 87'600 104:740 [1,2] 1 [1] 1 0'0 2024-04-25T08:17:23.969579+0000 0'0 2024-04-25T08:17:23.969579+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:26:19.732 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 200 0 200 0 0 2097152000 0 0 600 0 600 active+recovery_wait+undersized+degraded+remapped 2024-04-25T08:26:06.228296+0000 87'600 104:758 [1,2] 1 [1] 1 0'0 2024-04-25T08:17:17.811742+0000 0'0 2024-04-25T08:17:17.811742+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:26:19.732 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:26:19.732 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:26:19.733 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:26:19.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:19.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2024-04-25T08:26:19.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:188: TEST_recovery_priority: PRIO=190 2024-04-25T08:26:19.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:189: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T08:26:19.782 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:19.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:26:19.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-04-25T08:26:19.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:196: TEST_recovery_priority: ITEM=7.0 2024-04-25T08:26:19.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:197: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-04-25T08:26:19.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:19.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:26:19.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:202: TEST_recovery_priority: PRIO=255 2024-04-25T08:26:19.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:203: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:26:19.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: seq 1 10 2024-04-25T08:26:19.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:211: TEST_recovery_priority: for i in $(seq 1 $max_tries) 2024-04-25T08:26:19.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: grep -q 'doesn'\''t require recovery' 2024-04-25T08:26:19.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:213: TEST_recovery_priority: ceph pg force-recovery 3.0 2024-04-25T08:26:20.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:214: TEST_recovery_priority: break 2024-04-25T08:26:20.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:222: TEST_recovery_priority: sleep 2 2024-04-25T08:26:22.250 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:26:22.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:26:22.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:26:22.251 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:26:22.251 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:22.251 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:26:22.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:26:22.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:26:22.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:223: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:22.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:224: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:22.386 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:22.386 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:22.386 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:22.386 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:22.386 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:22.386 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:22.386 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:22.386 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:22.386 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 255, 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:22.387 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:22.388 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:22.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:22.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-04-25T08:26:22.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:225: TEST_recovery_priority: PRIO=255 2024-04-25T08:26:22.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:226: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:26:22.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:231: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:26:22.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:26:22.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:26:22.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:26:22.873 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:26:22.874 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:26:22.874 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:26:22.874 INFO:tasks.workunit.client.0.smithi134.stderr:4' 2024-04-25T08:26:22.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:26:22.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:22.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:26:23.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836614 2024-04-25T08:26:23.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836614 2024-04-25T08:26:23.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614' 2024-04-25T08:26:23.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:23.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:26:23.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673091 2024-04-25T08:26:23.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673091 2024-04-25T08:26:23.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673091' 2024-04-25T08:26:23.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:23.201 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:26:23.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509568 2024-04-25T08:26:23.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509568 2024-04-25T08:26:23.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673091 2-64424509568' 2024-04-25T08:26:23.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:23.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:26:23.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378749 2024-04-25T08:26:23.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378749 2024-04-25T08:26:23.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673091 2-64424509568 3-81604378749' 2024-04-25T08:26:23.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:23.511 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:26:23.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215226 2024-04-25T08:26:23.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215226 2024-04-25T08:26:23.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836614 1-42949673091 2-64424509568 3-81604378749 4-103079215226' 2024-04-25T08:26:23.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:23.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836614 2024-04-25T08:26:23.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:23.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:26:23.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836614 2024-04-25T08:26:23.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:23.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836614 2024-04-25T08:26:23.659 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836614 2024-04-25T08:26:23.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836614' 2024-04-25T08:26:23.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:24.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836611 -lt 21474836614 2024-04-25T08:26:24.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:26:25.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:26:25.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:25.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836614 -lt 21474836614 2024-04-25T08:26:25.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:25.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:25.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673091 2024-04-25T08:26:25.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:26:25.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673091 2024-04-25T08:26:25.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:25.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673091 2024-04-25T08:26:25.468 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673091 2024-04-25T08:26:25.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673091' 2024-04-25T08:26:25.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:26:25.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673091 -lt 42949673091 2024-04-25T08:26:25.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:25.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509568 2024-04-25T08:26:25.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:25.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:26:25.928 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509568 2024-04-25T08:26:25.928 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:25.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509568 2024-04-25T08:26:25.930 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509568 2024-04-25T08:26:25.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509568' 2024-04-25T08:26:25.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:26:26.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509568 -lt 64424509568 2024-04-25T08:26:26.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:26.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378749 2024-04-25T08:26:26.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:26.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:26:26.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378749 2024-04-25T08:26:26.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:26.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378749 2024-04-25T08:26:26.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378749' 2024-04-25T08:26:26.406 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378749 2024-04-25T08:26:26.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:26:26.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378749 -lt 81604378749 2024-04-25T08:26:26.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:26.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:26.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215226 2024-04-25T08:26:26.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:26:26.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215226 2024-04-25T08:26:26.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:26.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215226 2024-04-25T08:26:26.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215226' 2024-04-25T08:26:26.846 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215226 2024-04-25T08:26:26.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:26:27.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215226 -lt 103079215226 2024-04-25T08:26:27.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:235: TEST_recovery_priority: ceph pg cancel-force-recovery 7.0 2024-04-25T08:26:27.795 INFO:tasks.workunit.client.0.smithi134.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-recovery; 2024-04-25T08:26:27.795 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T08:26:27.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:236: TEST_recovery_priority: sleep 2 2024-04-25T08:26:29.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:26:29.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:26:29.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:26:29.814 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:26:29.814 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:29.814 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:26:29.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:26:29.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:26:29.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:238: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:29.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:239: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:29.947 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:29.947 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:29.947 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:29.947 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:29.947 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:29.947 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:29.947 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:29.947 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:29.947 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:29.948 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:29.949 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:29.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:29.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2024-04-25T08:26:29.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:240: TEST_recovery_priority: PRIO=190 2024-04-25T08:26:29.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:241: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T08:26:29.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:29.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:26:30.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: eval 'ITEM="3.0"' 2024-04-25T08:26:30.024 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:247: TEST_recovery_priority: ITEM=3.0 2024-04-25T08:26:30.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:248: TEST_recovery_priority: '[' 3.0 '!=' 3.0 ']' 2024-04-25T08:26:30.024 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:30.024 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:26:30.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:253: TEST_recovery_priority: PRIO=255 2024-04-25T08:26:30.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:254: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:26:30.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:261: TEST_recovery_priority: ceph pg cancel-force-recovery 3.0 2024-04-25T08:26:30.455 INFO:tasks.workunit.client.0.smithi134.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-recovery; 2024-04-25T08:26:30.455 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T08:26:30.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:262: TEST_recovery_priority: sleep 5 2024-04-25T08:26:35.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:26:35.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:26:35.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:26:35.475 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:26:35.475 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:35.475 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:26:35.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:26:35.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:26:35.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:263: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:35.591 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:35.593 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:35.594 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:35.594 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:35.594 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:35.594 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:35.594 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:266: TEST_recovery_priority: flush_pg_stats 2024-04-25T08:26:35.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:26:35.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:26:36.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:26:36.017 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:26:36.017 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:26:36.017 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:26:36.017 INFO:tasks.workunit.client.0.smithi134.stderr:4' 2024-04-25T08:26:36.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:26:36.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:36.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:26:36.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836618 2024-04-25T08:26:36.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836618 2024-04-25T08:26:36.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836618' 2024-04-25T08:26:36.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:36.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:26:36.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673095 2024-04-25T08:26:36.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673095 2024-04-25T08:26:36.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836618 1-42949673095' 2024-04-25T08:26:36.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:36.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:26:36.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509572 2024-04-25T08:26:36.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509572 2024-04-25T08:26:36.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836618 1-42949673095 2-64424509572' 2024-04-25T08:26:36.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:36.481 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:26:36.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378753 2024-04-25T08:26:36.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378753 2024-04-25T08:26:36.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836618 1-42949673095 2-64424509572 3-81604378753' 2024-04-25T08:26:36.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:36.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:26:36.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215230 2024-04-25T08:26:36.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215230 2024-04-25T08:26:36.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836618 1-42949673095 2-64424509572 3-81604378753 4-103079215230' 2024-04-25T08:26:36.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:36.779 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836618 2024-04-25T08:26:36.779 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:36.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:26:36.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836618 2024-04-25T08:26:36.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:36.781 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836618 2024-04-25T08:26:36.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836618 2024-04-25T08:26:36.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836618' 2024-04-25T08:26:36.782 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:37.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836616 -lt 21474836618 2024-04-25T08:26:37.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:26:38.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:26:38.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:38.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836619 -lt 21474836618 2024-04-25T08:26:38.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:38.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673095 2024-04-25T08:26:38.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:38.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:26:38.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673095 2024-04-25T08:26:38.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:38.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673095 2024-04-25T08:26:38.629 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673095 2024-04-25T08:26:38.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673095' 2024-04-25T08:26:38.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:26:39.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673096 -lt 42949673095 2024-04-25T08:26:39.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:39.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509572 2024-04-25T08:26:39.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:39.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:26:39.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509572 2024-04-25T08:26:39.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:39.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509572 2024-04-25T08:26:39.073 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509572 2024-04-25T08:26:39.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509572' 2024-04-25T08:26:39.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:26:39.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509573 -lt 64424509572 2024-04-25T08:26:39.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:39.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378753 2024-04-25T08:26:39.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:39.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:26:39.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378753 2024-04-25T08:26:39.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:39.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378753 2024-04-25T08:26:39.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378753' 2024-04-25T08:26:39.538 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378753 2024-04-25T08:26:39.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:26:39.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378754 -lt 81604378753 2024-04-25T08:26:39.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:39.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215230 2024-04-25T08:26:39.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:39.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:26:39.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215230 2024-04-25T08:26:39.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:39.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215230 2024-04-25T08:26:39.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215230' 2024-04-25T08:26:39.980 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215230 2024-04-25T08:26:39.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:26:40.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215231 -lt 103079215230 2024-04-25T08:26:40.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:267: TEST_recovery_priority: ceph pg force-recovery 7.0 2024-04-25T08:26:40.833 INFO:tasks.workunit.client.0.smithi134.stderr:instructing pg(s) [7.0] on osd.1 to force-recovery; 2024-04-25T08:26:40.833 INFO:tasks.workunit.client.0.smithi134.stderr: 2024-04-25T08:26:40.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:268: TEST_recovery_priority: sleep 2 2024-04-25T08:26:42.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:26:42.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:26:42.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:26:42.853 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:26:42.854 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:42.854 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:26:42.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:26:42.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:26:42.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:270: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:42.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:271: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:42.985 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:42.985 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:42.985 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:42.985 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:42.985 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:42.985 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:42.985 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:42.985 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:42.986 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:42.987 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:42.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:42.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2024-04-25T08:26:43.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:272: TEST_recovery_priority: PRIO=190 2024-04-25T08:26:43.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:273: TEST_recovery_priority: '[' 190 '!=' 190 ']' 2024-04-25T08:26:43.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:43.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: jq '.local_reservations.in_progress[0].item' 2024-04-25T08:26:43.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: eval 'ITEM="7.0"' 2024-04-25T08:26:43.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:279: TEST_recovery_priority: ITEM=7.0 2024-04-25T08:26:43.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:280: TEST_recovery_priority: '[' 7.0 '!=' 7.0 ']' 2024-04-25T08:26:43.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: cat td/osd-recovery-prio/out 2024-04-25T08:26:43.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: jq '.local_reservations.in_progress[0].prio' 2024-04-25T08:26:43.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:285: TEST_recovery_priority: PRIO=255 2024-04-25T08:26:43.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:286: TEST_recovery_priority: '[' 255 '!=' 255 ']' 2024-04-25T08:26:43.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:293: TEST_recovery_priority: ceph osd unset noout 2024-04-25T08:26:43.866 INFO:tasks.workunit.client.0.smithi134.stderr:noout is unset 2024-04-25T08:26:43.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:294: TEST_recovery_priority: ceph osd unset norecover 2024-04-25T08:26:44.792 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is unset 2024-04-25T08:26:44.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:26:44.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:26:44.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:26:44.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:26:44.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:26:44.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:26:44.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:26:44.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:296: TEST_recovery_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations' 2024-04-25T08:26:44.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations' 2024-04-25T08:26:44.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:26:44.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:26:44.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:26:44.821 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:26:44.821 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:26:44.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:26:44.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:26:44.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:26:44.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:26:44.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:26:44.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:26:44.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:26:44.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:26:44.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:26:45.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:26:45.375 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:26:45.376 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:26:45.376 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:26:45.376 INFO:tasks.workunit.client.0.smithi134.stderr:4' 2024-04-25T08:26:45.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:26:45.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:45.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:26:45.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836622 2024-04-25T08:26:45.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836622 2024-04-25T08:26:45.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836622' 2024-04-25T08:26:45.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:45.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:26:45.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673099 2024-04-25T08:26:45.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673099 2024-04-25T08:26:45.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836622 1-42949673099' 2024-04-25T08:26:45.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:45.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:26:45.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509576 2024-04-25T08:26:45.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509576 2024-04-25T08:26:45.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836622 1-42949673099 2-64424509576' 2024-04-25T08:26:45.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:45.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:26:46.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378757 2024-04-25T08:26:46.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378757 2024-04-25T08:26:46.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836622 1-42949673099 2-64424509576 3-81604378757' 2024-04-25T08:26:46.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:26:46.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:26:46.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215234 2024-04-25T08:26:46.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215234 2024-04-25T08:26:46.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836622 1-42949673099 2-64424509576 3-81604378757 4-103079215234' 2024-04-25T08:26:46.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:46.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836622 2024-04-25T08:26:46.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:46.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:26:46.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836622 2024-04-25T08:26:46.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:46.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836622 2024-04-25T08:26:46.190 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836622 2024-04-25T08:26:46.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836622' 2024-04-25T08:26:46.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:26:46.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836622 -lt 21474836622 2024-04-25T08:26:46.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:46.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:46.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673099 2024-04-25T08:26:46.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:26:46.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673099 2024-04-25T08:26:46.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:46.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673099 2024-04-25T08:26:46.630 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673099 2024-04-25T08:26:46.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673099' 2024-04-25T08:26:46.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:26:47.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673099 -lt 42949673099 2024-04-25T08:26:47.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:47.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:47.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509576 2024-04-25T08:26:47.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:26:47.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509576 2024-04-25T08:26:47.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:47.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509576 2024-04-25T08:26:47.066 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509576 2024-04-25T08:26:47.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509576' 2024-04-25T08:26:47.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:26:47.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509576 -lt 64424509576 2024-04-25T08:26:47.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:47.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378757 2024-04-25T08:26:47.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:47.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:26:47.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378757 2024-04-25T08:26:47.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:47.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378757 2024-04-25T08:26:47.743 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378757 2024-04-25T08:26:47.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378757' 2024-04-25T08:26:47.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:26:48.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378757 -lt 81604378757 2024-04-25T08:26:48.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:26:48.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:26:48.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215234 2024-04-25T08:26:48.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:26:48.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215234 2024-04-25T08:26:48.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:26:48.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215234 2024-04-25T08:26:48.859 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215234 2024-04-25T08:26:48.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215234' 2024-04-25T08:26:48.860 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:26:49.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215235 -lt 103079215234 2024-04-25T08:26:49.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:26:49.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:26:49.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:26:50.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 3 == 0 2024-04-25T08:26:50.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:26:50.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:26:50.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:26:50.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:26:50.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:26:50.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:26:50.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:26:50.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:26:50.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:26:50.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:26:50.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:26:51.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:26:51.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:26:51.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:26:51.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:26:51.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:51.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:26:51.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:51.530 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:51.530 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:51.531 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:51.532 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:51.533 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:51.533 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:51.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:26:51.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:26:51.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:26:51.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:26:51.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:26:51.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:26:51.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:26:51.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:26:51.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:26:52.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:26:52.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:26:52.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:26:52.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:26:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:26:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:26:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:26:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:26:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:26:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:26:53.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:26:53.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:26:53.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:26:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T08:26:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T08:26:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T08:26:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:53.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:26:53.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:54.019 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:54.020 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:54.021 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:54.021 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:54.021 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:54.021 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:54.021 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:54.021 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:54.021 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:54.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T08:26:54.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:26:54.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:26:54.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:26:54.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:26:54.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:26:54.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:26:54.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:26:54.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:26:54.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:26:54.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:26:54.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:26:54.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:26:55.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:26:55.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:26:55.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:26:55.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:26:55.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:26:55.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:26:55.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:26:55.701 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:26:55.701 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:26:56.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8748692 2024-04-25T08:26:56.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8748692 '!=' null 2024-04-25T08:26:56.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:26:56.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:56.557 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:26:56.557 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:56.661 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:56.661 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:56.661 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:56.661 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:56.661 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:56.661 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:56.662 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:56.663 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:56.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:26:56.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:26:56.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:26:56.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:26:56.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:26:56.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:26:56.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:26:56.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:26:56.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:26:57.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:26:57.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:26:57.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:26:57.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:26:57.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:26:57.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:26:57.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:26:57.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:26:57.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:26:57.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:26:57.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:26:57.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:26:57.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:26:58.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7492371 2024-04-25T08:26:58.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7492371 '!=' null 2024-04-25T08:26:58.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:26:58.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:58.481 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:26:58.481 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:26:58.594 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:26:58.595 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:26:58.595 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:58.595 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:58.595 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:26:58.595 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:58.595 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:26:58.595 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:26:58.596 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:26:58.597 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:26:58.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:26:58.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:26:58.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:26:58.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:26:58.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:26:58.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:26:58.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:26:58.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:26:58.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:26:59.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:26:59.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:26:59.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:26:59.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:26:59.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:26:59.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:26:59.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:26:59.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:26:59.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:26:59.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:26:59.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:26:59.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:26:59.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:00.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14849966 2024-04-25T08:27:00.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14849966 '!=' null 2024-04-25T08:27:00.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:00.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:00.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:00.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:00.933 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:00.934 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:00.935 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:00.935 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:00.935 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:00.935 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:00.935 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:00.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:01.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:01.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:01.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:01.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:01.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:01.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:01.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:01.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:01.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:01.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:01.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:01.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:02.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:02.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:02.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850147 2024-04-25T08:27:02.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850147 '!=' null 2024-04-25T08:27:02.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:02.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:02.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:02.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:03.081 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:03.081 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:03.081 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:03.081 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:03.081 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:03.082 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:03.083 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:03.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:03.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:03.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:03.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:03.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:03.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:03.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:03.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:03.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:03.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:03.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:03.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:03.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:04.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:04.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:04.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:04.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:04.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:04.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:04.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:04.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:04.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:05.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21838399 2024-04-25T08:27:05.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21838399 '!=' null 2024-04-25T08:27:05.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:05.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:05.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:05.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:05.428 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:05.429 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:05.430 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:05.430 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:05.430 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:05.430 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:05.430 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:05.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:05.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:05.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:05.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:05.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:05.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:05.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:05.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:05.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:05.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:05.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:05.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:05.964 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:06.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:06.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:06.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:06.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:06.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:06.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:06.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:06.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:06.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:07.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850113 2024-04-25T08:27:07.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850113 '!=' null 2024-04-25T08:27:07.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:07.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:07.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:07.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:07.514 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:07.514 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:07.514 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:07.514 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:07.514 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:07.514 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:07.515 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:07.516 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:07.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:07.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:07.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:07.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:07.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:07.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:07.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:07.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:07.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:08.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:08.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:08.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:08.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:09.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:09.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:09.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:09.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:09.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:09.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:09.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:09.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:09.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:09.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712216 2024-04-25T08:27:09.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712216 '!=' null 2024-04-25T08:27:09.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:09.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:09.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:09.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:09.893 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:09.893 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:09.893 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:09.893 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:09.893 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:09.893 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:09.894 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:09.895 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:09.896 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:09.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:10.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:10.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:10.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:10.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:10.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:10.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:10.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:10.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:10.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:10.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:10.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:10.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:11.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:11.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:11.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:11.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:11.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:11.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:11.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:11.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:11.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:12.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850314 2024-04-25T08:27:12.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850314 '!=' null 2024-04-25T08:27:12.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:12.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:12.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:12.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:12.206 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:12.206 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:12.206 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:12.206 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:12.206 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:12.206 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:12.206 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:12.206 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:12.206 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:12.207 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:12.208 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:12.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:12.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:12.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:12.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:12.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:12.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:12.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:12.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:12.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:12.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:12.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:12.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:12.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:13.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:13.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:13.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:13.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:13.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:13.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:13.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:13.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:13.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:14.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850336 2024-04-25T08:27:14.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850336 '!=' null 2024-04-25T08:27:14.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:14.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:14.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:14.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:14.414 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:14.414 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:14.415 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:14.416 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:14.417 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:14.417 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:14.417 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:14.417 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:14.417 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:14.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:14.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:14.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:14.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:14.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:14.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:14.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:14.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:14.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:14.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:14.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:14.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:14.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:15.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:15.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:15.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:15.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:15.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:15.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:15.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:15.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:15.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:16.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723849 2024-04-25T08:27:16.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723849 '!=' null 2024-04-25T08:27:16.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:16.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:16.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:16.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:16.739 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:16.740 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:16.741 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:16.742 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:16.742 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:16.742 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:16.742 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:16.742 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:16.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:16.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:16.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:16.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:16.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:16.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:16.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:16.855 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:16.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:17.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:17.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:17.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:17.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:18.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:18.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:18.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:18.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:18.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:18.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:18.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:18.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:18.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:19.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712258 2024-04-25T08:27:19.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712258 '!=' null 2024-04-25T08:27:19.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:19.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:19.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:19.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:19.823 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:19.823 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:19.823 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:19.823 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:19.823 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:19.823 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:19.823 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:19.824 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:19.825 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:19.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:19.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:19.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:19.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:19.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:19.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:19.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:19.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:19.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:20.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:20.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:20.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:20.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:21.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:21.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:21.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:21.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:21.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:21.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:21.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:21.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:21.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:21.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850224 2024-04-25T08:27:21.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850224 '!=' null 2024-04-25T08:27:21.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:21.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:21.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:21.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:22.054 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:22.054 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:22.055 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:22.056 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:22.057 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:22.057 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:22.057 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:22.057 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:22.057 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:22.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:22.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:22.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:22.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:22.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:22.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:22.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:22.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:22.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:22.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:22.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:22.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:22.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:23.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:23.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:23.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:23.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:23.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:23.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:23.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:23.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:23.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:24.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850217 2024-04-25T08:27:24.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850217 '!=' null 2024-04-25T08:27:24.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:24.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:24.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:24.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:24.193 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:24.193 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:24.193 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:24.194 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:24.195 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:24.196 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:24.196 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:24.196 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:24.196 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:24.196 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:24.196 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:24.196 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:24.196 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:24.196 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:24.197 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:24.197 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:24.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:24.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:24.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:24.308 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:24.308 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:24.308 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:24.308 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:24.308 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:24.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:24.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:24.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:24.726 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:24.726 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:25.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:25.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:25.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:25.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:25.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:25.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:25.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:25.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:25.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:26.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712196 2024-04-25T08:27:26.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712196 '!=' null 2024-04-25T08:27:26.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:26.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:26.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:26.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:26.594 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:26.595 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:26.596 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:26.596 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:26.596 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:26.596 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:26.596 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:26.596 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:26.596 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:26.596 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:26.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:26.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:26.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:26.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:26.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:26.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:26.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:26.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:26.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:27.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:27.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:27.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:27.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:28.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:28.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:28.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:28.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:28.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:28.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:28.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:28.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:28.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:29.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713091 2024-04-25T08:27:29.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713091 '!=' null 2024-04-25T08:27:29.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:29.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:29.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:29.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:29.332 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:29.333 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:29.334 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:29.334 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:29.334 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:29.334 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:29.334 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:29.334 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:29.334 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:29.334 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:29.334 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:29.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:29.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:29.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:29.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:29.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:29.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:29.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:29.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:29.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:29.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:29.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:29.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:29.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:30.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:30.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:30.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:30.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:30.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:30.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:30.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:30.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:30.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:31.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724597 2024-04-25T08:27:31.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724597 '!=' null 2024-04-25T08:27:31.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:31.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:31.326 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:31.326 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:31.434 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:31.434 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:31.435 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:31.436 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:31.437 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:31.437 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:31.437 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:31.437 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:31.437 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:31.437 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:31.437 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:31.437 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:31.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:31.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:31.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:31.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:31.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:31.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:31.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:31.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:31.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:32.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:32.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:32.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:32.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:33.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:33.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:33.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:33.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:33.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:33.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:33.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:33.413 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:33.413 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:34.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724769 2024-04-25T08:27:34.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724769 '!=' null 2024-04-25T08:27:34.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:34.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:34.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:34.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:34.284 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:34.284 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:34.284 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:34.284 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:34.285 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:34.286 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:34.287 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:34.287 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:34.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:34.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:34.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:34.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:34.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:34.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:34.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:34.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:34.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:35.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:35.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:35.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:35.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:35.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:35.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:35.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:35.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:35.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:35.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:35.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:35.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:35.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:36.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713307 2024-04-25T08:27:36.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713307 '!=' null 2024-04-25T08:27:36.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:36.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:36.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:36.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:36.789 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:36.789 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:36.789 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:36.790 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:36.790 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:36.790 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:36.790 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:36.790 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:36.790 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:36.791 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:36.791 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:36.791 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:36.791 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:36.791 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:36.791 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:36.792 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:36.792 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:36.792 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:36.792 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:36.792 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:36.792 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:36.793 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:36.793 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:36.793 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:36.793 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:36.793 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:36.793 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:36.794 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:36.794 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:36.794 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:36.794 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:36.794 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:36.794 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:36.795 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:36.795 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:36.795 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:36.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:36.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:36.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:36.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:36.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:36.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:36.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:36.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:36.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:37.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:37.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:37.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:37.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:38.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:38.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:38.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:38.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:38.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:38.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:38.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:38.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:38.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:39.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712736 2024-04-25T08:27:39.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712736 '!=' null 2024-04-25T08:27:39.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:39.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:39.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:39.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:39.573 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:39.573 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:39.573 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:39.573 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:39.573 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:39.574 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:39.575 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:39.576 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:39.576 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:39.576 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:39.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:39.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:39.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:39.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:39.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:39.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:39.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:39.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:39.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:40.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:40.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:40.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:40.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:40.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:40.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:40.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:40.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:40.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:40.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:40.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:40.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:40.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:41.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850494 2024-04-25T08:27:41.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850494 '!=' null 2024-04-25T08:27:41.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:41.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:41.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:41.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:42.062 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:42.062 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:42.062 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:42.062 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:42.062 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:42.062 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:42.063 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:42.064 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:42.065 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:42.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:42.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:42.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:42.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:42.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:42.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:42.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:42.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:42.178 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:42.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:42.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:42.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:42.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:43.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:43.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:43.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:43.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:43.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:43.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:43.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:43.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:43.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:44.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850375 2024-04-25T08:27:44.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850375 '!=' null 2024-04-25T08:27:44.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:44.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:44.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:44.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:44.219 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:44.219 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:44.219 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:44.219 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:44.220 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:44.221 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:44.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:44.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:44.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:44.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:44.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:44.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:44.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:44.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:44.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:44.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:44.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:44.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:44.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:45.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:45.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:45.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:45.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:45.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:45.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:45.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:45.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:45.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:46.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712133 2024-04-25T08:27:46.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712133 '!=' null 2024-04-25T08:27:46.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:46.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:46.601 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:46.601 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:46.712 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:46.712 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:46.712 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:46.712 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:46.712 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:46.712 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:46.712 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:46.713 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:46.714 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:46.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:46.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:46.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:46.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:46.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:46.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:46.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:46.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:46.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:47.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:47.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:47.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:47.250 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:47.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:47.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:48.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712383 2024-04-25T08:27:48.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712383 '!=' null 2024-04-25T08:27:48.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:48.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:48.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:48.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:48.921 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:48.980 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:48.981 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:48.982 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:48.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:49.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:49.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:49.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:49.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:49.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:49.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:49.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:49.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:49.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:49.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:49.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:49.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:50.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:50.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:50.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:50.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:50.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:50.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:50.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:50.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:50.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:51.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850552 2024-04-25T08:27:51.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850552 '!=' null 2024-04-25T08:27:51.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:51.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:51.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:51.899 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:52.009 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:52.009 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:52.009 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:52.009 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:52.009 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:52.009 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:52.010 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:52.011 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:52.012 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:52.012 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:52.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:52.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:52.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:52.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:52.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:52.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:52.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:52.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:52.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:52.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:52.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:52.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:52.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:53.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:53.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:53.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:53.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:53.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:53.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:53.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:53.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:53.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:54.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850241 2024-04-25T08:27:54.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850241 '!=' null 2024-04-25T08:27:54.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:54.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:54.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:54.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:54.708 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:54.709 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:54.710 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:54.711 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:54.711 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:54.711 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:54.711 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:54.711 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:54.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:54.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:54.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:54.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:54.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:54.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:54.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:54.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:54.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:55.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:55.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:55.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:55.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:55.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:55.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:55.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:55.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:55.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:55.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:55.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:55.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:55.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723918 2024-04-25T08:27:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723918 '!=' null 2024-04-25T08:27:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:56.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:56.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:56.886 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:56.886 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:56.887 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:56.888 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:56.889 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:56.889 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:56.889 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:56.889 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:56.889 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:56.889 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:56.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:27:56.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:27:56.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:27:57.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:27:57.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:27:57.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:27:57.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:27:57.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:27:57.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:27:57.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:27:57.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:27:57.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:27:57.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:27:58.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:27:58.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:27:58.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:27:58.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:27:58.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:27:58.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:27:58.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:27:58.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:27:58.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:27:59.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712326 2024-04-25T08:27:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712326 '!=' null 2024-04-25T08:27:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:27:59.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:59.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:27:59.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:27:59.954 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:27:59.955 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:27:59.956 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:27:59.957 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:27:59.957 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:27:59.957 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:27:59.957 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:27:59.957 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:27:59.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:00.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:00.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:00.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:00.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:00.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:00.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:00.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:00.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:00.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:00.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:00.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:00.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:01.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:01.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:01.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:01.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:01.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:01.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:01.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:01.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:01.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:02.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723876 2024-04-25T08:28:02.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723876 '!=' null 2024-04-25T08:28:02.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:02.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:02.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:02.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:02.540 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:02.541 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:02.542 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:02.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:02.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:02.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:02.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:02.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:02.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:02.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:02.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:02.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:03.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:03.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:03.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:03.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:03.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:03.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:03.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:03.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:03.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:03.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:03.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:03.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:03.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:04.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723816 2024-04-25T08:28:04.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723816 '!=' null 2024-04-25T08:28:04.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:04.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:04.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:04.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:04.823 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:04.824 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:04.825 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:04.826 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:04.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:04.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:04.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:04.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:04.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:04.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:04.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:04.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:04.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:05.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:05.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:05.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:05.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:06.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:06.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:06.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:06.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:06.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:06.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:06.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:06.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:06.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:07.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850814 2024-04-25T08:28:07.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850814 '!=' null 2024-04-25T08:28:07.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:07.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:07.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:07.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:07.287 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:07.287 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:07.287 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:07.288 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:07.289 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:07.290 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:07.290 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:07.290 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:07.290 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:07.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:07.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:07.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:07.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:07.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:07.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:07.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:07.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:07.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:07.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:07.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:07.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:07.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:08.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:08.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:08.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:08.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:08.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:08.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:08.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:08.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:08.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:09.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22712532 2024-04-25T08:28:09.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22712532 '!=' null 2024-04-25T08:28:09.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:09.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:09.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:09.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:09.117 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:09.118 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:09.119 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:09.120 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:09.120 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:09.120 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:09.120 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:09.120 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:09.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:09.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:09.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:09.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:09.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:09.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:09.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:09.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:09.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:09.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:09.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:09.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:09.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:10.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:10.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:10.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:10.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:10.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:10.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:10.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:10.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:10.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:11.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724031 2024-04-25T08:28:11.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724031 '!=' null 2024-04-25T08:28:11.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:11.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:11.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:11.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:11.247 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:11.247 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:11.248 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:11.249 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:11.250 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:11.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:11.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:11.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:11.362 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:11.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:11.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:11.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:11.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:11.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:11.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:11.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:11.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:11.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:12.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:12.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:12.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:12.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:12.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:12.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:12.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:12.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:12.462 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:13.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723855 2024-04-25T08:28:13.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723855 '!=' null 2024-04-25T08:28:13.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:13.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:13.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:13.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:13.276 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:13.276 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:13.276 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:13.276 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:13.276 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:13.276 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:13.277 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:13.278 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:13.279 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:13.279 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:13.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:13.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:13.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:13.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:13.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:13.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:13.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:13.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:13.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:13.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:13.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:13.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:13.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:14.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:14.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:14.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:14.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:14.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:14.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:14.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:14.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:14.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:15.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20965116 2024-04-25T08:28:15.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20965116 '!=' null 2024-04-25T08:28:15.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:15.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:15.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:15.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:15.126 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:15.126 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:15.126 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:15.126 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:15.126 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:15.126 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:15.127 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:15.128 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:15.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:15.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:15.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:15.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:15.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:15.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:15.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:15.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:15.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:15.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:15.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:15.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:15.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:16.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:16.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:16.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:16.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:16.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:16.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:16.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:16.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:16.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:17.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13102904 2024-04-25T08:28:17.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13102904 '!=' null 2024-04-25T08:28:17.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:17.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:17.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:17.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:17.284 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:17.284 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:17.284 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:17.285 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:17.286 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:17.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:17.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:17.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:17.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:17.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:17.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:17.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:17.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:17.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:17.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:17.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:17.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:17.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:18.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:18.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:18.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:18.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:18.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:18.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:18.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:18.463 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:18.463 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:19.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19217245 2024-04-25T08:28:19.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19217245 '!=' null 2024-04-25T08:28:19.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:19.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:19.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:19.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:19.273 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:19.274 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:19.275 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:19.275 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:19.275 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:19.275 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:19.275 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:19.275 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:19.275 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:19.275 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:19.275 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:19.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:19.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:19.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:19.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:19.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:19.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:19.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:19.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:19.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:19.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:19.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:19.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:19.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:20.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:20.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:20.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:20.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:20.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:20.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:20.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:20.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:20.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:21.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11355750 2024-04-25T08:28:21.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11355750 '!=' null 2024-04-25T08:28:21.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:21.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:21.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:21.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:21.249 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:21.249 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:21.249 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:21.249 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:21.249 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:21.249 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:21.249 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:21.249 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:21.250 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:21.251 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:21.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:21.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:21.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:21.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:21.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:21.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:21.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:21.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:21.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:21.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:21.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:21.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:21.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:22.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:22.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:22.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:22.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:22.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:22.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:22.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:22.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:22.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:22.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11355959 2024-04-25T08:28:22.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11355959 '!=' null 2024-04-25T08:28:22.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:22.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:22.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:22.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:23.084 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:23.084 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:23.084 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:23.084 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:23.084 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:23.085 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:23.086 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:23.087 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:23.087 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:23.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:23.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:23.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:23.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:23.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:23.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:23.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:23.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:23.201 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:23.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:23.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:23.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:23.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:24.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:24.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:24.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:24.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:24.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:24.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:24.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:24.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:24.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:25.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471120 2024-04-25T08:28:25.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471120 '!=' null 2024-04-25T08:28:25.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:25.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:25.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:25.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:25.115 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:25.115 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:25.115 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:25.115 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:25.116 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:25.117 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:25.118 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:25.118 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:25.118 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:25.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:25.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:25.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:25.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:25.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:25.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:25.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:25.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:25.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:25.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:25.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:25.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:25.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:26.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:26.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:26.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:26.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:26.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:26.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:26.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:26.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:26.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:27.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229898 2024-04-25T08:28:27.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229898 '!=' null 2024-04-25T08:28:27.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:27.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:27.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:27.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:27.412 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:27.413 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:27.414 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:27.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:27.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:27.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:27.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:27.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:27.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:27.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:27.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:27.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:27.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:27.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:27.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:27.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:28.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:28.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:28.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:28.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:28.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:28.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:28.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:28.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:28.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:29.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471381 2024-04-25T08:28:29.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471381 '!=' null 2024-04-25T08:28:29.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:29.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:29.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:29.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:29.254 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:29.255 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:29.256 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:29.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:29.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:29.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:29.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:29.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:29.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:29.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:29.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:29.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:29.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:29.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:29.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:29.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:30.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:30.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:31.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356363 2024-04-25T08:28:31.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356363 '!=' null 2024-04-25T08:28:31.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:31.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:31.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:31.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:31.431 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:31.431 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:31.431 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:31.431 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:31.431 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:31.431 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:31.431 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:31.431 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:31.431 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:31.432 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:31.433 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:31.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:31.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:31.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:31.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:31.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:31.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:31.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:31.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:31.547 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:32.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:32.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:32.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:32.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:32.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:32.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:32.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:32.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:32.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:32.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:32.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:32.906 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:32.906 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:33.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356391 2024-04-25T08:28:33.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356391 '!=' null 2024-04-25T08:28:33.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:33.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:33.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:33.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:33.718 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:33.718 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:33.718 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:33.718 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:33.718 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:33.718 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:33.719 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:33.720 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:33.721 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:33.721 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:33.721 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:33.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:33.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:33.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:33.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:33.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:33.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:33.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:33.834 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:33.834 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:34.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:34.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:34.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:34.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:35.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:35.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:35.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:35.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:35.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:35.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:35.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:35.204 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:35.204 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:35.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471101 2024-04-25T08:28:35.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471101 '!=' null 2024-04-25T08:28:35.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:35.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:35.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:35.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:35.933 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:35.933 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:35.933 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:35.933 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:35.933 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:35.933 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:35.933 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:35.933 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:35.934 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:35.935 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:35.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:36.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:36.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:36.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:36.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:36.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:36.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:36.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:36.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:36.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:36.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:36.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:36.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:37.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:37.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:37.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:37.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:37.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:37.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:37.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:37.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:37.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:37.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356002 2024-04-25T08:28:37.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356002 '!=' null 2024-04-25T08:28:37.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:37.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:37.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:37.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:37.917 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:37.918 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:37.919 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:37.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:38.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:38.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:38.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:38.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:38.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:38.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:38.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:38.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:38.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:38.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:38.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:38.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:39.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:39.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:39.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:39.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:39.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:39.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:39.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:39.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:39.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:39.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597053 2024-04-25T08:28:39.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597053 '!=' null 2024-04-25T08:28:39.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:39.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:39.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:39.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:40.023 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:40.023 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:40.023 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:40.023 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:40.024 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:40.025 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:40.026 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:40.026 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:40.026 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:40.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:40.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:40.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:40.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:40.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:40.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:40.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:40.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:40.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:40.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:40.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:40.576 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:40.576 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:41.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:41.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:41.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:41.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:41.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:41.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:41.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:41.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:41.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:41.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11355796 2024-04-25T08:28:41.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11355796 '!=' null 2024-04-25T08:28:41.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:41.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:41.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:41.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:41.941 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:41.941 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:41.941 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:41.941 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:41.942 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:41.943 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:41.944 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:41.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:42.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:42.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:42.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:42.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:42.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:42.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:42.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:42.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:42.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:42.541 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:42.541 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:42.541 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:43.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:43.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:43.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:43.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:43.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:43.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:43.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:43.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:43.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:43.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11355859 2024-04-25T08:28:43.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11355859 '!=' null 2024-04-25T08:28:43.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:43.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:43.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:43.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:43.903 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:43.903 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:43.903 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:43.903 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:43.903 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:43.903 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:43.903 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:43.904 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:43.905 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:43.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:44.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:44.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:44.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:44.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:44.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:44.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:44.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:44.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:44.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:44.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:44.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:44.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:44.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:44.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:44.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:44.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:44.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:44.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:44.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:44.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:44.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:45.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597238 2024-04-25T08:28:45.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597238 '!=' null 2024-04-25T08:28:45.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:45.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:45.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:45.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:45.705 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:45.706 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:45.707 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:45.707 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:45.707 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:45.707 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:45.707 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:45.707 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:45.707 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:45.707 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:45.707 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:45.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:45.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:45.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:45.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:45.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:45.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:45.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:45.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:45.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:46.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:46.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:46.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:46.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:46.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:46.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:46.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:46.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:46.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:46.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:46.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:46.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:46.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:47.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10482675 2024-04-25T08:28:47.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10482675 '!=' null 2024-04-25T08:28:47.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:47.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:47.481 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:47.481 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:47.588 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:47.588 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:47.588 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:47.588 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:47.588 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:47.588 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:47.588 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:47.588 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:47.588 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:47.589 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:47.590 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:47.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:47.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:47.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:47.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:47.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:47.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:47.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:47.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:47.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:48.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:48.110 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:48.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:48.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:48.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:48.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:48.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:48.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:48.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:48.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:48.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:48.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:48.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:49.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724427 2024-04-25T08:28:49.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724427 '!=' null 2024-04-25T08:28:49.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:49.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:49.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:49.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:49.480 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:49.480 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:49.480 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:49.481 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:49.482 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:49.483 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:49.483 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:49.483 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:49.483 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:49.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:49.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:49.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:49.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:49.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:49.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:49.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:49.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:49.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:50.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:50.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:50.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:50.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:50.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:50.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:50.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:50.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:50.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:50.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:50.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:50.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:50.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:51.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10482972 2024-04-25T08:28:51.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10482972 '!=' null 2024-04-25T08:28:51.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:51.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:51.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:51.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:51.329 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:51.330 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:51.331 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:51.332 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:51.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:51.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:51.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:51.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:51.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:51.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:51.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:51.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:51.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:51.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:51.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:51.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:51.933 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:52.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:52.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:52.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:52.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:52.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:52.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:52.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:52.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:52.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:53.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10483250 2024-04-25T08:28:53.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10483250 '!=' null 2024-04-25T08:28:53.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:53.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:53.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:53.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:53.213 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:53.214 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "7.0", 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 255, 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:53.215 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:53.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:53.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:53.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:53.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:53.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:53.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:53.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:53.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:53.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:53.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:53.743 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:53.743 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:53.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:54.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:54.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:54.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:54.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:54.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:54.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:54.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:54.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:54.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:54.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598717 2024-04-25T08:28:54.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598717 '!=' null 2024-04-25T08:28:54.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:54.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:54.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:54.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:55.065 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:55.065 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:55.065 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:55.065 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:55.065 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:55.065 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:55.065 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:55.065 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:55.066 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:55.067 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:55.067 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:55.067 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:55.067 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:55.067 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:55.067 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:55.067 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:55.067 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:55.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:55.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:55.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:55.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:55.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:55.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:55.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:55.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:55.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:55.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:55.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:55.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:55.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:56.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:56.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:56.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:56.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:56.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:56.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:56.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:56.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:56.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:57.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13628300 2024-04-25T08:28:57.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13628300 '!=' null 2024-04-25T08:28:57.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:57.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:57.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:57.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:57.676 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:28:57.676 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:28:57.676 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:57.676 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:57.676 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:28:57.676 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:57.676 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:28:57.676 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:28:57.676 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:28:57.677 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:28:57.678 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:28:57.678 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:28:57.678 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:28:57.678 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:28:57.678 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:28:57.678 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:28:57.678 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:28:57.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:28:57.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:28:57.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:28:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:28:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:28:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:28:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:28:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:28:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:28:58.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:28:58.208 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:28:58.209 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:28:58.209 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:28:59.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:28:59.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:28:59.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:28:59.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:28:59.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:28:59.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:28:59.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:28:59.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:28:59.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:28:59.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724731 2024-04-25T08:28:59.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724731 '!=' null 2024-04-25T08:28:59.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:28:59.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:28:59.987 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:28:59.987 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:00.099 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:00.099 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:00.100 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:00.101 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:00.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:00.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:00.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:00.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:00.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:00.215 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:00.215 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:00.215 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:00.215 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:00.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:29:00.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:00.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:00.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:29:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:29:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:01.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:01.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:02.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724088 2024-04-25T08:29:02.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724088 '!=' null 2024-04-25T08:29:02.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:02.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:02.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:02.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:02.640 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:02.640 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:02.640 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:02.640 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:02.640 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:02.640 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:02.640 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:02.641 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:02.642 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:02.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:02.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:02.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:02.756 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:02.756 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:02.756 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:02.756 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:02.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:02.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:03.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:29:03.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:03.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:03.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:03.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 3 2024-04-25T08:29:03.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:29:03.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:03.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:03.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:03.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:03.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:03.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:03.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:04.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6551533 2024-04-25T08:29:04.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6551533 '!=' null 2024-04-25T08:29:04.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:04.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:04.665 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:04.665 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:04.765 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:04.765 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:04.765 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:04.765 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:04.766 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:04.767 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:04.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:04.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:04.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:04.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:04.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:04.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:04.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:04.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:04.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:05.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:05.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:05.683 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:05.683 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:06.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:06.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 0 2024-04-25T08:29:06.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:29:06.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=1 2024-04-25T08:29:06.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:06.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:06.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:06.676 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:06.676 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:06.676 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:06.676 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:06.676 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:06.676 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:06.676 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:06.676 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:06.676 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:06.677 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:06.678 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:06.678 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:06.678 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:06.678 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:06.678 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:06.678 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:06.678 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:06.678 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:06.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:06.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:06.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:06.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:06.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:06.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:06.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:06.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:06.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:07.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:07.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:07.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:07.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:08.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:08.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:08.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:08.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:08.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:08.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:08.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:08.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:08.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:09.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19830828 2024-04-25T08:29:09.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19830828 '!=' null 2024-04-25T08:29:09.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:09.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:09.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:09.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:09.205 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:09.205 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:09.205 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:09.205 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:09.205 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:09.206 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:09.207 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:09.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:09.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:09.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:09.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:09.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:09.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:09.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:09.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:09.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:09.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:09.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:09.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:09.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:10.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:10.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:10.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:10.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:10.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:10.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:10.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:10.675 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:10.675 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:11.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976718 2024-04-25T08:29:11.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976718 '!=' null 2024-04-25T08:29:11.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:11.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:11.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:11.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:11.463 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:11.464 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:11.465 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:11.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:11.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:11.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:11.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:11.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:11.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:11.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:11.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:11.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:12.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:12.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:12.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:12.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:12.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:12.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:12.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:12.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:12.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:12.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:12.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:12.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:12.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:13.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13976887 2024-04-25T08:29:13.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13976887 '!=' null 2024-04-25T08:29:13.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:13.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:13.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:13.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:13.638 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:13.639 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:13.640 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:13.641 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:13.641 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:13.641 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:13.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:13.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:13.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:13.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:13.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:13.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:13.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:13.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:13.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:14.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:14.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:14.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:14.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:14.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:14.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:14.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:14.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:15.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=21839280 2024-04-25T08:29:15.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 21839280 '!=' null 2024-04-25T08:29:15.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:15.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:15.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:15.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:15.848 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:15.848 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:15.849 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:15.850 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:15.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:15.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:15.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:15.964 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:15.964 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:15.964 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:15.964 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:15.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:15.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:16.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:16.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:16.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:16.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:17.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:17.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:17.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:17.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:17.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:17.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:17.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:17.076 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:17.076 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:17.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14850549 2024-04-25T08:29:17.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14850549 '!=' null 2024-04-25T08:29:17.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:17.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:17.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:17.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:17.827 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:17.827 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:17.827 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:17.827 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:17.827 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:17.827 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:17.827 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:17.828 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:17.829 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:17.829 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:17.829 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:17.829 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:17.829 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:17.829 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:17.829 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:17.829 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:17.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:17.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:17.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:17.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:17.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:17.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:17.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:17.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:17.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:18.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:18.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:18.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:18.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:19.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:19.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:19.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:19.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:19.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:19.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:19.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:19.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:19.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:19.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20091750 2024-04-25T08:29:19.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20091750 '!=' null 2024-04-25T08:29:19.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:19.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:19.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:19.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:19.869 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:19.869 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:19.869 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:19.869 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:19.870 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:19.871 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:19.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:19.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:19.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:19.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:19.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:19.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:19.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:19.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:19.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:20.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:20.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:20.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:20.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:21.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:21.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:21.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:21.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:21.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:21.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:21.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:21.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:21.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:21.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13103553 2024-04-25T08:29:21.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13103553 '!=' null 2024-04-25T08:29:21.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:21.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:21.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:21.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:21.726 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:21.727 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:21.728 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:21.728 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:21.728 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:21.728 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:21.728 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:21.728 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:21.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:21.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:21.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:21.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:21.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:21.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:21.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:22.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:22.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:22.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:22.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:22.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:22.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:22.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:22.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:22.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:22.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:22.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:22.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:22.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:23.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13103606 2024-04-25T08:29:23.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13103606 '!=' null 2024-04-25T08:29:23.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:23.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:23.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:23.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:23.639 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:23.639 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:23.639 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:23.639 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:23.639 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:23.639 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:23.640 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:23.641 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:23.642 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:23.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:23.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:23.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:23.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:23.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:23.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:23.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:23.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:23.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:24.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:24.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:24.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:24.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:24.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:24.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:24.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:24.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:24.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:24.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:24.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:24.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:24.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:25.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=19218549 2024-04-25T08:29:25.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 19218549 '!=' null 2024-04-25T08:29:25.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:25.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:25.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:25.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:25.736 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:25.737 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:25.738 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:25.738 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:25.738 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:25.738 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:25.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:25.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:25.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:25.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:25.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:25.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:25.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:25.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:25.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:26.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:26.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:26.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:26.270 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:27.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:27.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:27.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:27.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:27.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:27.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:27.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:27.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:27.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:27.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356151 2024-04-25T08:29:27.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356151 '!=' null 2024-04-25T08:29:27.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:27.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:27.699 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:27.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:27.810 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:27.811 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:27.812 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:27.812 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:27.812 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:27.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:27.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:27.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:27.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:27.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:27.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:27.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:27.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:27.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:28.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:28.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:28.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:28.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:29.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:29.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:29.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:29.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:29.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:29.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:29.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:29.035 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:29.035 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:29.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597453 2024-04-25T08:29:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597453 '!=' null 2024-04-25T08:29:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:29.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:29.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:29.836 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:29.837 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:29.838 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:29.838 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:29.838 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:29.838 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:29.838 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:29.838 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:29.838 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:29.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:29.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:29.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:29.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:29.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:29.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:29.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:29.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:29.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:30.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:30.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:30.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:30.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:31.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:31.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:31.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:31.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:31.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:31.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:31.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:31.167 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:31.167 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:31.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356349 2024-04-25T08:29:31.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356349 '!=' null 2024-04-25T08:29:31.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:31.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:31.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:31.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:31.861 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:31.861 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:31.861 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:31.861 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:31.861 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:31.861 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:31.861 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:31.861 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:31.861 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:31.862 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:31.863 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:31.863 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:31.863 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:31.863 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:31.863 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:31.863 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:31.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:31.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:31.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:31.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:31.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:31.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:31.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:31.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:31.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:32.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:32.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:32.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:32.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:33.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:33.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:33.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:33.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:33.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:33.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:33.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:33.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:33.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:33.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356370 2024-04-25T08:29:33.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356370 '!=' null 2024-04-25T08:29:33.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:33.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:33.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:33.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:33.982 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:33.982 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:33.982 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:33.982 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:33.982 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:33.983 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:33.984 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:33.984 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:33.984 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:33.984 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:33.984 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:33.984 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:33.984 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:33.984 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:33.984 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:33.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:34.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:34.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:34.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:34.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:34.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:34.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:34.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:34.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:34.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:34.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:34.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:34.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:35.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:35.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:35.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:35.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:35.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:35.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:35.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:35.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:35.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:35.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597721 2024-04-25T08:29:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597721 '!=' null 2024-04-25T08:29:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:35.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:35.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:35.894 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:35.895 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:35.896 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:35.897 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:35.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:36.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:36.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:36.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:36.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:36.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:36.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:36.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:36.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:36.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:36.459 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:36.459 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:36.459 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:37.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:37.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:37.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:37.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:37.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:37.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:37.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:37.093 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:37.093 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:37.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10482761 2024-04-25T08:29:37.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10482761 '!=' null 2024-04-25T08:29:37.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:37.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:37.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:37.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:37.743 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:37.744 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:37.745 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:37.745 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:37.745 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:37.745 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:37.745 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:37.745 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:37.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:37.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:37.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:37.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:37.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:37.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:37.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:37.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:37.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:38.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:38.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:38.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:38.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:38.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:38.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:38.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:38.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:38.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:38.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:38.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:38.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:38.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:39.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15724477 2024-04-25T08:29:39.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15724477 '!=' null 2024-04-25T08:29:39.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:39.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:39.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:39.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:39.783 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:39.783 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:39.783 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:39.783 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:39.783 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:39.783 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:39.783 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:39.783 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:39.783 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:39.784 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:39.785 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:39.785 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:39.785 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:39.785 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:39.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:39.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:39.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:39.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:39.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:39.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:39.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:39.899 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:39.899 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:40.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:40.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:40.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:40.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:41.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:41.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:41.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:41.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:41.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:41.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:41.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:41.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:41.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:41.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10482958 2024-04-25T08:29:41.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10482958 '!=' null 2024-04-25T08:29:41.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:41.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:41.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:41.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:41.958 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:41.958 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:41.958 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:41.958 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:41.959 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:41.960 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:41.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:42.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:42.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:42.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:42.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:42.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:42.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:42.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:42.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:42.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:42.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:42.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:42.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:43.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:43.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:43.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10482771 2024-04-25T08:29:43.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10482771 '!=' null 2024-04-25T08:29:43.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:43.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:43.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:43.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:43.792 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:43.793 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:43.794 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:43.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:43.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:43.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:43.908 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:43.908 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:43.908 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:43.908 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:43.908 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:43.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:44.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:44.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:44.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:44.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:45.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:45.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:45.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:45.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:45.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:45.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:45.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:45.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:45.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:45.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597624 2024-04-25T08:29:45.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597624 '!=' null 2024-04-25T08:29:45.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:45.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:45.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:45.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:45.763 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:45.763 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:45.763 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:45.763 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:45.763 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:45.763 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:45.763 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:45.764 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:45.765 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:45.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:45.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:45.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:45.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:45.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:45.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:45.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:45.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:45.878 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:46.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:46.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:46.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:46.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:46.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:46.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:46.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:46.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:46.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:46.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:46.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:46.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:46.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:47.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356215 2024-04-25T08:29:47.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356215 '!=' null 2024-04-25T08:29:47.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:47.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:47.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:47.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:47.557 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:47.557 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:47.557 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:47.557 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:47.557 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:47.557 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:47.557 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:47.557 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:47.558 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:47.559 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:47.559 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:47.559 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:47.559 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:47.559 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:47.559 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:47.559 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:47.559 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:47.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:47.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:47.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:47.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:47.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:47.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:47.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:47.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:47.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:48.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:48.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:48.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:48.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:48.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:48.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:48.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:48.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:48.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:48.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:48.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:48.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:48.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:49.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597563 2024-04-25T08:29:49.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597563 '!=' null 2024-04-25T08:29:49.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:49.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:49.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:49.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:49.425 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:49.426 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:49.427 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:49.427 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:49.427 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:49.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:49.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:49.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:49.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:49.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:49.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:49.540 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:49.541 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:49.541 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:49.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:49.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:49.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:49.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:50.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:50.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:50.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:50.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:50.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:50.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:50.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:50.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:50.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:51.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356105 2024-04-25T08:29:51.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356105 '!=' null 2024-04-25T08:29:51.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:51.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:51.078 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:51.078 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:51.186 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:51.187 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:51.188 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:51.189 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:51.189 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:51.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:51.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:51.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:51.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:51.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:51.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:51.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:51.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:51.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:51.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:51.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:51.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:51.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:52.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:52.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:53.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356447 2024-04-25T08:29:53.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356447 '!=' null 2024-04-25T08:29:53.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:53.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:53.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:53.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:53.324 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:53.324 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:53.325 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:53.326 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:53.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:53.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:53.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:53.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:53.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:53.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:53.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:53.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:53.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:53.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:53.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:53.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:53.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:54.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:54.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:54.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:54.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:54.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:54.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:54.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:54.541 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:54.541 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:55.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471389 2024-04-25T08:29:55.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471389 '!=' null 2024-04-25T08:29:55.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:55.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:55.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:55.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:55.621 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:55.621 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:55.621 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:55.621 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:55.621 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:55.621 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:55.621 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:55.622 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:55.623 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:55.623 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:55.623 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:55.623 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:55.623 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:55.623 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:55.623 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:55.623 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:55.623 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:55.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:55.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:55.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:55.736 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:55.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:55.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:55.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:55.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:55.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:56.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:56.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:56.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:56.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:56.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:56.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:57.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356313 2024-04-25T08:29:57.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356313 '!=' null 2024-04-25T08:29:57.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:57.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:57.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:57.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:57.638 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:57.638 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:57.638 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:57.638 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:57.638 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:57.638 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:57.639 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:57.640 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:57.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:57.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:57.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:57.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:57.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:57.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:57.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:57.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:29:57.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:58.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:29:58.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:29:58.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:29:58.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:29:58.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:29:58.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:29:58.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:29:58.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:29:58.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:29:58.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:29:58.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:29:58.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:29:58.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:29:59.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597666 2024-04-25T08:29:59.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597666 '!=' null 2024-04-25T08:29:59.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:29:59.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:59.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:29:59.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:29:59.506 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:29:59.506 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:29:59.506 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:59.506 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:29:59.507 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:29:59.508 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:29:59.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:29:59.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:29:59.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:29:59.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:29:59.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:29:59.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:29:59.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:29:59.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:29:59.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:00.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:00.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:00.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:00.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:00.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:00.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:00.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:00.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:00.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:00.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:00.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:00.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:00.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:01.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356273 2024-04-25T08:30:01.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356273 '!=' null 2024-04-25T08:30:01.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:01.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:01.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:01.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:01.387 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:01.387 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:01.387 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:01.387 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:01.387 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:01.387 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:01.387 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:01.388 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:01.389 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:01.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:01.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:01.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:01.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:01.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:01.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:01.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:01.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:01.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:01.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:01.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:01.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:01.918 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:02.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:02.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:02.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:02.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:02.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:02.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:02.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:02.549 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:02.549 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:03.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356313 2024-04-25T08:30:03.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356313 '!=' null 2024-04-25T08:30:03.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:03.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:03.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:03.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:03.385 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:03.386 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:03.387 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:03.387 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:03.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:03.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:03.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:03.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:03.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:03.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:03.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:03.501 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:03.501 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:03.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:03.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:03.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:03.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:04.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:04.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:04.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:04.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:04.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:04.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:04.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:04.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:04.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:05.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470947 2024-04-25T08:30:05.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470947 '!=' null 2024-04-25T08:30:05.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:05.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:05.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:05.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:05.296 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:05.297 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:05.298 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:05.299 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:05.299 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:05.299 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:05.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:05.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:05.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:05.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:05.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:05.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:05.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:05.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:05.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:05.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:05.806 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:05.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:05.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:06.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:06.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:06.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:06.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:06.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:06.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:06.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:06.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:06.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:07.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356224 2024-04-25T08:30:07.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356224 '!=' null 2024-04-25T08:30:07.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:07.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:07.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:07.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:07.614 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:07.614 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:07.614 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:07.615 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:07.616 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:07.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:07.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:07.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:07.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:07.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:07.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:07.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:07.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:07.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:08.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:08.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:08.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:08.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:09.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:09.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:09.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:09.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:09.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:09.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:09.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:09.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:09.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:09.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597539 2024-04-25T08:30:09.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597539 '!=' null 2024-04-25T08:30:09.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:09.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:09.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:09.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:09.738 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:09.738 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:09.738 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:09.738 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:09.738 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:09.738 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:09.738 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:09.739 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:09.740 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:09.740 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:09.740 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:09.740 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:09.740 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:09.740 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:09.740 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:09.740 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:09.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:09.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:09.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:09.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:09.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:09.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:09.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:09.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:09.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:10.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:10.275 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:10.276 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:10.276 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:10.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:10.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:10.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:10.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:10.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:10.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:10.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:10.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:10.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:11.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356507 2024-04-25T08:30:11.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356507 '!=' null 2024-04-25T08:30:11.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:11.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:11.559 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:11.559 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:11.666 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:11.666 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:11.666 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:11.666 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:11.666 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:11.666 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:11.666 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:11.666 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:11.666 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:11.667 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:11.668 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:11.668 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:11.668 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:11.668 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:11.668 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:11.668 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:11.668 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:11.668 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:11.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:11.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:11.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:11.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:11.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:11.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:11.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:11.782 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:11.782 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:12.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:12.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:12.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:12.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:12.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:12.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:12.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:12.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:12.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:12.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:12.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:12.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:12.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:13.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356478 2024-04-25T08:30:13.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356478 '!=' null 2024-04-25T08:30:13.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:13.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:13.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:13.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:13.646 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:13.647 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:13.648 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:13.648 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:13.648 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:13.648 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:13.648 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:13.648 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:13.648 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:13.648 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:13.648 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:13.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:13.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:13.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:13.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:13.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:13.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:13.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:13.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:13.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:14.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:14.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:14.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:14.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:14.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:14.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:14.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:14.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:14.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:14.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:14.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:14.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:14.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:15.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597869 2024-04-25T08:30:15.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597869 '!=' null 2024-04-25T08:30:15.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:15.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:15.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:15.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:15.539 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:15.539 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:15.539 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:15.540 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:15.541 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:15.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:15.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:15.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:15.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:15.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:15.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:15.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:15.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:15.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:16.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:16.126 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:16.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:16.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:16.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:16.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:17.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10482903 2024-04-25T08:30:17.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10482903 '!=' null 2024-04-25T08:30:17.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:17.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:17.468 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:17.468 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:17.576 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:17.577 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:17.578 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:17.578 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:17.578 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:17.578 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:17.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:17.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:17.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:17.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:17.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:17.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:17.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:17.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:17.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:18.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:18.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:18.090 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:18.090 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:18.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:18.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:18.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:18.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:18.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:18.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:18.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:18.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:18.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597882 2024-04-25T08:30:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597882 '!=' null 2024-04-25T08:30:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:19.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:19.494 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:19.595 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:19.596 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:19.597 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:19.598 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:19.598 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:19.598 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:19.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:19.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:19.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:19.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:19.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:19.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:19.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:19.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:19.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:20.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:20.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:20.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:20.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:20.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:20.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:20.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:20.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:20.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:20.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:20.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:20.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:20.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:21.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356533 2024-04-25T08:30:21.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356533 '!=' null 2024-04-25T08:30:21.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:21.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:21.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:21.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:21.923 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:21.923 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:21.923 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:21.924 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:21.925 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:21.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:22.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:22.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:22.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:22.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:22.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:22.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:22.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:22.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:22.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:22.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:22.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:22.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:23.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:23.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:23.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:23.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:23.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:23.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:23.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:23.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:23.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:23.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356331 2024-04-25T08:30:23.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356331 '!=' null 2024-04-25T08:30:23.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:23.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:23.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:23.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:24.047 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:24.047 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:24.047 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:24.047 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:24.048 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:24.049 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:24.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:24.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:24.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:24.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:24.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:24.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:24.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:24.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:24.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:24.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:24.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:24.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:24.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:25.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:25.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:25.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:25.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:25.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:25.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:25.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:25.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:25.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:26.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597799 2024-04-25T08:30:26.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597799 '!=' null 2024-04-25T08:30:26.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:26.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:26.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:26.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:26.116 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:26.116 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:26.116 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:26.117 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:26.118 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:26.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:26.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:26.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:26.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:26.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:26.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:26.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:26.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:26.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:26.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:26.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:26.637 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:26.637 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:27.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:27.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:27.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:27.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:27.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:27.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:27.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:27.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:27.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:28.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356491 2024-04-25T08:30:28.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356491 '!=' null 2024-04-25T08:30:28.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:28.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:28.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:28.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:28.232 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:28.232 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:28.232 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:28.232 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:28.232 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:28.233 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:28.234 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:28.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:28.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:28.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:28.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:28.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:28.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:28.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:28.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:28.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:28.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:28.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:28.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:28.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:29.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:29.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:29.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:29.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:29.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:29.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:29.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:29.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:29.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:30.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471321 2024-04-25T08:30:30.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471321 '!=' null 2024-04-25T08:30:30.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:30.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:30.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:30.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:30.170 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:30.170 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:30.170 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:30.170 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:30.170 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:30.170 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:30.171 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:30.172 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:30.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:30.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:30.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:30.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:30.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:30.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:30.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:30.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:30.287 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:30.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:30.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:30.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:30.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:31.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:31.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:31.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:31.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:31.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:31.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:31.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:31.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:31.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:32.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356305 2024-04-25T08:30:32.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356305 '!=' null 2024-04-25T08:30:32.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:32.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:32.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:32.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:32.140 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:32.140 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:32.140 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:32.140 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:32.140 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:32.140 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:32.140 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:32.141 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:32.141 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:32.141 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:32.141 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:32.141 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:32.141 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:32.141 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:32.141 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:32.142 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:32.142 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:32.142 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:32.142 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:32.142 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:32.142 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:32.142 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:32.142 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:32.142 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:32.143 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:32.143 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:32.143 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:32.143 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:32.143 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:32.143 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:32.143 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:32.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:32.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:32.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:32.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:32.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:32.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:32.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:32.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:32.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:32.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:32.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:32.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:32.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:33.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:33.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:33.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:33.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:33.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:33.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:33.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:33.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:33.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:34.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356415 2024-04-25T08:30:34.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356415 '!=' null 2024-04-25T08:30:34.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:34.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:34.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:34.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:34.468 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:34.468 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:34.468 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:34.468 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:34.468 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:34.468 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:34.468 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:34.469 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:34.470 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:34.470 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:34.470 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:34.470 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:34.470 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:34.470 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:34.470 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:34.470 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:34.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:34.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:34.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:34.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:34.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:34.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:34.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:34.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:34.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:35.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:35.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:35.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:35.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:35.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:35.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:35.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:35.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:35.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:35.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:35.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:35.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:35.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:36.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597779 2024-04-25T08:30:36.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597779 '!=' null 2024-04-25T08:30:36.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:36.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:36.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:36.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:36.438 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:36.439 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:36.440 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:36.440 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:36.440 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:36.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:36.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:36.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:36.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:36.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:37.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:37.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:37.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:37.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:37.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:37.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:37.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:37.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:37.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:37.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:37.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:37.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:37.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:38.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471112 2024-04-25T08:30:38.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471112 '!=' null 2024-04-25T08:30:38.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:38.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:38.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:38.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:38.692 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:38.693 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:38.694 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:38.694 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:38.694 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:38.694 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:38.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:38.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:38.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:38.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:38.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:38.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:38.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:38.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:38.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:39.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:39.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:39.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:39.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:39.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:39.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:39.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:39.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:39.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:39.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:39.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:39.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:39.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:40.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471112 2024-04-25T08:30:40.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471112 '!=' null 2024-04-25T08:30:40.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:40.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:40.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:40.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:40.732 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:40.733 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:40.734 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:40.735 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:40.735 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:40.735 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:40.735 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:40.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:40.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:40.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:40.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:40.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:40.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:40.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:40.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:40.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:41.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:41.292 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:41.292 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:41.293 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:41.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:41.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:41.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:41.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:41.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:41.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:41.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:41.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:41.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:42.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356209 2024-04-25T08:30:42.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356209 '!=' null 2024-04-25T08:30:42.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:42.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:42.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:42.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:42.671 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:42.672 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:42.673 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:42.673 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:42.673 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:42.673 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:42.673 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:42.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:42.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:42.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:42.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:42.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:42.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:42.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:42.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:42.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:43.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:43.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:43.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:43.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:43.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:43.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:43.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:43.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:43.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:43.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:43.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:43.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:43.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:44.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470736 2024-04-25T08:30:44.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470736 '!=' null 2024-04-25T08:30:44.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:44.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:44.480 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:44.480 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:44.575 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:44.575 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:44.575 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:44.576 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:44.577 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:44.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:44.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:44.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:44.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:44.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:44.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:44.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:44.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:44.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:45.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:45.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:45.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:45.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:45.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:45.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:45.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:45.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:45.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:45.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:45.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:45.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:45.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470736 2024-04-25T08:30:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470736 '!=' null 2024-04-25T08:30:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:46.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:46.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:46.500 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:46.501 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:46.502 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:46.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:46.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:46.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:46.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:46.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:46.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:46.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:46.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:46.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:47.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:47.079 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:47.079 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:47.079 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:47.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:47.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:47.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:47.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:47.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:47.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:47.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:47.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:47.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:48.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229501 2024-04-25T08:30:48.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229501 '!=' null 2024-04-25T08:30:48.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:48.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:48.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:48.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:48.331 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:48.331 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:48.331 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:48.331 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:48.331 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:48.331 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:48.331 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:48.331 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:48.331 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:48.332 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:48.333 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:48.333 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:48.333 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:48.333 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:48.333 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:48.333 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:48.333 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:48.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:48.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:48.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:48.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:48.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:48.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:48.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:48.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:48.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:48.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:48.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:48.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:48.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:49.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:49.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:50.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344157 2024-04-25T08:30:50.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344157 '!=' null 2024-04-25T08:30:50.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:50.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:50.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:50.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:50.223 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:50.223 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:50.223 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:50.223 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:50.224 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:50.225 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:50.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:50.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:50.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:50.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:50.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:50.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:50.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:50.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:50.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:50.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:50.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:50.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:50.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:51.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:51.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:51.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:51.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:51.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:51.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:51.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:51.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:51.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:51.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229412 2024-04-25T08:30:51.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229412 '!=' null 2024-04-25T08:30:51.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:51.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:51.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:51.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:51.959 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:51.960 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:51.961 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:51.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:52.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:52.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:52.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:52.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:52.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:52.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:52.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:52.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:52.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:52.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:52.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:52.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:53.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:53.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:53.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:53.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:53.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:53.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:53.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:53.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:53.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:53.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229721 2024-04-25T08:30:53.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229721 '!=' null 2024-04-25T08:30:53.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:53.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:53.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:53.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:54.065 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:54.066 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:54.067 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:54.067 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:54.067 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:54.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:54.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:54.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:54.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:54.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:54.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:54.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:54.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:54.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:54.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:54.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:54.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:54.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:55.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:55.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:55.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:56.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344447 2024-04-25T08:30:56.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344447 '!=' null 2024-04-25T08:30:56.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:56.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:56.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:56.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:56.163 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:56.163 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:56.163 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:56.163 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:56.163 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:56.164 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:56.165 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:56.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:56.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:56.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:56.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:56.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:56.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:56.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:56.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:56.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:56.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:56.702 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:56.702 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:56.702 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:57.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:57.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:57.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:57.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:57.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:57.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:57.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:57.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:30:57.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:58.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229590 2024-04-25T08:30:58.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229590 '!=' null 2024-04-25T08:30:58.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:30:58.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:58.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:30:58.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:30:58.196 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:30:58.196 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:30:58.196 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:58.196 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:58.196 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:30:58.197 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:30:58.198 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:30:58.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:30:58.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:30:58.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:30:58.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:30:58.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:30:58.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:30:58.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:30:58.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:30:58.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:30:58.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:30:58.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:30:58.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:30:58.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:30:59.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:30:59.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:30:59.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:30:59.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:30:59.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:30:59.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:30:59.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:30:59.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:30:59.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:00.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344418 2024-04-25T08:31:00.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344418 '!=' null 2024-04-25T08:31:00.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:00.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:00.156 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:00.156 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:00.252 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:00.253 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:00.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:00.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:00.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:00.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:00.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:00.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:00.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:00.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:00.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:00.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:00.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:00.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:00.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:01.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:01.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:01.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:01.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:01.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:01.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:01.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:01.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:01.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:02.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229648 2024-04-25T08:31:02.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229648 '!=' null 2024-04-25T08:31:02.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:02.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:02.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:02.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:02.184 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:02.185 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:02.186 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:02.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:02.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:02.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:02.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:02.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:02.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:02.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:02.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:02.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:02.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:02.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:02.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:02.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:03.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:03.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:03.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:03.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:03.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:03.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:03.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:03.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:03.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:03.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229598 2024-04-25T08:31:03.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229598 '!=' null 2024-04-25T08:31:03.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:03.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:03.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:03.899 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:04.006 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:04.006 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:04.007 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:04.008 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:04.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:04.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:04.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:04.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:04.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:04.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:04.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:04.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:04.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:04.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:04.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:04.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:04.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:05.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:05.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:05.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:05.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:05.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:05.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:05.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:05.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:05.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344417 2024-04-25T08:31:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344417 '!=' null 2024-04-25T08:31:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:05.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:05.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:05.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:05.910 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:05.911 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:05.912 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:05.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:06.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:06.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:06.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:06.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:06.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:06.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:06.027 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:06.027 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:06.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:06.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:06.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:06.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:07.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:07.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:07.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:07.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:07.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:07.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:07.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:07.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:07.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:07.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229626 2024-04-25T08:31:07.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229626 '!=' null 2024-04-25T08:31:07.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:07.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:07.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:07.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:07.880 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:07.880 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:07.880 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:07.880 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:07.881 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:07.882 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:07.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:07.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:07.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:07.995 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:07.995 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:07.995 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:07.995 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:07.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:07.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:08.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:08.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:08.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:08.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:09.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:09.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:09.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:09.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:09.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:09.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:09.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:09.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:09.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:09.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471198 2024-04-25T08:31:09.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471198 '!=' null 2024-04-25T08:31:09.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:09.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:09.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:09.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:10.034 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:10.034 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:10.034 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:10.034 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:10.034 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:10.034 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:10.035 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:10.036 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:10.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:10.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:10.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:10.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:10.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:10.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:10.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:10.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:10.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:10.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:10.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:10.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:10.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:11.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:11.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:11.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:11.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:11.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:11.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:11.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:11.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:11.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:11.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356242 2024-04-25T08:31:11.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356242 '!=' null 2024-04-25T08:31:11.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:11.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:11.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:11.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:11.846 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:11.847 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:11.848 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:11.849 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:11.849 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:11.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:11.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:11.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:11.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:11.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:11.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:11.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:11.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:11.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:12.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:12.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:12.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:12.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:13.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:13.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:13.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:13.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:13.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:13.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:13.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:13.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:13.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:13.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356471 2024-04-25T08:31:13.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356471 '!=' null 2024-04-25T08:31:13.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:13.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:13.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:13.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:14.025 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:14.025 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:14.025 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:14.025 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:14.025 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:14.025 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:14.025 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:14.026 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:14.027 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:14.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:14.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:14.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:14.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:14.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:14.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:14.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:14.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:14.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:14.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:14.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:14.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:14.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:15.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:15.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:15.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:15.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:15.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:15.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:15.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:15.361 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:15.361 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:16.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471622 2024-04-25T08:31:16.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471622 '!=' null 2024-04-25T08:31:16.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:16.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:16.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:16.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:16.194 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:16.194 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:16.194 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:16.194 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:16.194 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:16.194 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:16.194 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:16.194 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:16.195 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:16.196 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:16.196 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:16.196 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:16.196 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:16.196 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:16.196 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:16.196 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:16.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:16.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:16.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:16.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:16.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:16.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:16.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:16.310 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:16.310 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:16.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:16.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:16.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:16.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:17.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:17.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:17.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:17.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:17.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:17.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:17.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:17.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:17.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:17.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356307 2024-04-25T08:31:17.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356307 '!=' null 2024-04-25T08:31:17.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:17.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:17.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:17.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:18.102 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:18.103 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:18.104 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:18.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:18.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:18.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:18.218 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:18.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:18.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:18.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:18.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:18.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:18.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:18.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:18.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:18.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:19.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:19.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:19.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:19.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:19.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:19.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:19.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:19.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:19.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:19.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356531 2024-04-25T08:31:19.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356531 '!=' null 2024-04-25T08:31:19.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:19.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:19.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:19.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:20.034 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:20.034 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:20.034 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:20.035 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:20.036 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:20.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:20.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:20.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:20.149 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:20.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:20.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:20.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:20.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:20.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:20.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:20.664 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:20.665 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:20.665 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:21.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:21.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:21.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:21.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:21.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:21.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:21.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:21.204 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:21.204 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:21.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229917 2024-04-25T08:31:21.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229917 '!=' null 2024-04-25T08:31:21.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:21.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:21.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:21.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:21.947 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:21.948 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:21.949 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:21.949 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:21.949 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:21.949 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:21.949 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:21.949 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:21.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:22.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:22.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:22.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:22.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:22.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:22.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:22.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:22.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:22.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:22.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:22.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:22.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:23.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:23.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:23.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:23.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:23.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:23.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:23.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:23.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:23.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:24.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12229928 2024-04-25T08:31:24.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12229928 '!=' null 2024-04-25T08:31:24.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:24.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:24.120 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:24.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:24.230 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:24.230 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:24.230 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:24.230 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:24.231 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [ 2024-04-25T08:31:24.231 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:24.231 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 190, 2024-04-25T08:31:24.231 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T08:31:24.231 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:24.231 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:24.232 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:24.232 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:24.232 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:24.232 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:24.232 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:24.232 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T08:31:24.233 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:24.233 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:24.233 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "1.0", 2024-04-25T08:31:24.233 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:24.233 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:24.233 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:24.233 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:24.234 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:24.234 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:24.234 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:24.234 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:24.234 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:24.234 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:24.235 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:24.235 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:24.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:24.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:24.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:24.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:24.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:24.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:24.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:24.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:24.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:24.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:24.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:24.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:24.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:25.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:25.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:25.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:25.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:25.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:25.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:25.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:25.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:25.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:26.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=18344721 2024-04-25T08:31:26.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 18344721 '!=' null 2024-04-25T08:31:26.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:26.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:26.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:26.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:26.923 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:26.923 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:26.923 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:26.923 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:26.923 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:26.924 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:26.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:27.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:27.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:27.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:27.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:27.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:27.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:27.040 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:27.040 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:27.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:27.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:27.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:27.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:28.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:28.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:28.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:28.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:28.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:28.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:28.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:28.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:28.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:29.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14675967 2024-04-25T08:31:29.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14675967 '!=' null 2024-04-25T08:31:29.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:29.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:29.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:29.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:29.242 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:29.243 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:29.244 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:29.244 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:29.244 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:29.244 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:29.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:29.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:29.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:29.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:29.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:29.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:29.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:29.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:29.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:31:29.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:29.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:29.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:31.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 3 2024-04-25T08:31:31.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 1 '!=' 1 2024-04-25T08:31:31.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:31.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:31.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:31.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:31.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:31.021 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:31.021 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:32.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17034453 2024-04-25T08:31:32.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17034453 '!=' null 2024-04-25T08:31:32.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:32.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:32.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:32.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:32.325 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:32.325 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:32.325 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:32.326 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:32.327 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:32.327 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:32.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:32.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:32.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:32.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:32.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:32.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:32.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:32.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:32.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:33.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:33.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:33.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:33.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:33.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:33.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 1 2024-04-25T08:31:33.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:31:33.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-04-25T08:31:33.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:33.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:33.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:34.053 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:34.053 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:34.053 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:34.053 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:34.053 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:34.053 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:34.053 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:34.053 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:34.054 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:34.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:34.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:34.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:34.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:34.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:34.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:34.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:34.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:34.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:35.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:35.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:35.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:35.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:35.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:35.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:35.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:35.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:35.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:35.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:35.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:35.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:35.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:36.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20874493 2024-04-25T08:31:36.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20874493 '!=' null 2024-04-25T08:31:36.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:36.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:36.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:36.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:36.536 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:36.536 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:36.536 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:36.536 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:36.536 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:36.536 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:36.536 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:36.536 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:36.537 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:36.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:36.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:36.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:36.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:36.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:36.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:36.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:36.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:36.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:37.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:37.143 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:37.143 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:37.143 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:38.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:38.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:38.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:38.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:38.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:38.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:38.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:38.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:38.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:38.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16176111 2024-04-25T08:31:38.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16176111 '!=' null 2024-04-25T08:31:38.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:38.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:38.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:38.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:39.065 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:39.065 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:39.065 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:39.066 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:39.067 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:39.067 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:39.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:39.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:39.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:39.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:39.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:39.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:39.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:39.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:39.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:39.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:39.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:39.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:39.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:40.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:40.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:40.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:40.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:40.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:40.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:40.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:40.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:40.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:41.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25332159 2024-04-25T08:31:41.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25332159 '!=' null 2024-04-25T08:31:41.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:41.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:41.154 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:41.154 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:41.256 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:41.257 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:41.258 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:41.258 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:41.258 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:41.258 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:41.258 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:41.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:41.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:41.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:41.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:41.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:41.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:41.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:41.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:41.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:41.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:41.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:41.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:41.808 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:42.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:42.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:42.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:42.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:42.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:42.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:42.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:42.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:42.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:43.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20091076 2024-04-25T08:31:43.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20091076 '!=' null 2024-04-25T08:31:43.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:43.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:43.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:43.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:43.524 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:43.524 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:43.524 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:43.524 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:43.524 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:43.525 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:43.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:43.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:43.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:43.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:43.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:43.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:43.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:43.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:43.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:44.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:44.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:44.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:44.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:45.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:45.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:45.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=27953227 2024-04-25T08:31:45.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 27953227 '!=' null 2024-04-25T08:31:45.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:45.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:45.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:45.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:45.984 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:45.984 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:45.984 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:45.984 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:45.984 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:45.984 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:45.985 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:46.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:46.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:46.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:46.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:46.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:46.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:46.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:46.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:46.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:46.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:46.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:46.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:46.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:47.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:47.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:47.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:47.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:47.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:47.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:47.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:47.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:47.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:48.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17470785 2024-04-25T08:31:48.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17470785 '!=' null 2024-04-25T08:31:48.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:48.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:48.333 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:48.333 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:48.435 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:48.435 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:48.435 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:48.435 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:48.435 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:48.435 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:48.435 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:48.436 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:48.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:48.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:48.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:48.549 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:48.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:48.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:48.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:48.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:48.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:48.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:48.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:48.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:48.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:49.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:49.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:49.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:49.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:49.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:49.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:49.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:49.567 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:49.567 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:50.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471108 2024-04-25T08:31:50.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471108 '!=' null 2024-04-25T08:31:50.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:50.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:50.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:50.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:50.289 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:50.290 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:50.290 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:50.290 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:50.290 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:50.290 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:50.290 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:50.290 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:50.290 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:50.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:50.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:50.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:50.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:50.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:50.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:50.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:50.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:50.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:50.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:50.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:50.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:50.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:51.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:51.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:51.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:51.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:51.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:51.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:51.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:51.718 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:51.718 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:52.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=25333098 2024-04-25T08:31:52.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 25333098 '!=' null 2024-04-25T08:31:52.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:52.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:52.561 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:52.561 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:52.666 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:52.667 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:52.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:52.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:52.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:52.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:52.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:52.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:52.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:52.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:52.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:53.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:53.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:53.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:53.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:54.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:54.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:54.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:55.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459750 2024-04-25T08:31:55.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459750 '!=' null 2024-04-25T08:31:55.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:55.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:55.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:55.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:55.164 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:55.165 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:55.165 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:55.165 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:55.165 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:55.165 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:55.165 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:55.165 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:55.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:55.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:55.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:55.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:55.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:55.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:55.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:55.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:55.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:55.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:55.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:55.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:55.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:56.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:56.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:56.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:56.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:56.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:56.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:56.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:56.706 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:56.706 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:57.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597461 2024-04-25T08:31:57.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597461 '!=' null 2024-04-25T08:31:57.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:57.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:57.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:57.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:57.930 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:31:57.930 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:31:57.931 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:31:57.932 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:31:57.932 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:31:57.932 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:31:57.932 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:31:57.932 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:31:57.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:31:58.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:31:58.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:31:58.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:31:58.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:31:58.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:31:58.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:31:58.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:31:58.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:31:58.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:31:58.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:31:58.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:31:58.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:31:59.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:31:59.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:31:59.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:31:59.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:31:59.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:31:59.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:31:59.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:31:59.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:31:59.297 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:31:59.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597492 2024-04-25T08:31:59.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597492 '!=' null 2024-04-25T08:31:59.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:31:59.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:31:59.973 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:31:59.973 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:00.079 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:00.079 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:00.079 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:00.079 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:00.079 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:00.079 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:00.079 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:00.080 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:00.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:00.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:00.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:00.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:00.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:00.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:00.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:00.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:00.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:00.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:00.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:00.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:00.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:01.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:01.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:01.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:01.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:01.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:01.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:01.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:01.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:01.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:02.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597355 2024-04-25T08:32:02.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597355 '!=' null 2024-04-25T08:32:02.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:02.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:02.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:02.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:02.934 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:02.935 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:02.935 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:02.935 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:02.935 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:02.935 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:02.935 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:02.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:03.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:03.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:03.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:03.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:03.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:03.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:03.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:03.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:03.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:03.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:03.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:03.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:04.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:04.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:04.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:04.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:04.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:04.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:04.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:04.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:04.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:04.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597355 2024-04-25T08:32:04.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597355 '!=' null 2024-04-25T08:32:04.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:04.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:04.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:04.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:04.991 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:04.992 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:04.993 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:04.993 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:04.993 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:04.993 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:04.993 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:05.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:05.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:05.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:05.107 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:05.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:05.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:05.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:05.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:05.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:05.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:05.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:05.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:05.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:06.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:06.716 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:06.716 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:07.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723686 2024-04-25T08:32:07.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723686 '!=' null 2024-04-25T08:32:07.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:07.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:07.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:07.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:07.528 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:07.529 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:07.529 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:07.529 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:07.529 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:07.529 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:07.529 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:07.529 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:07.529 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:07.529 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:07.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:07.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:07.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:07.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:07.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:07.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:07.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:07.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:07.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:08.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:08.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:08.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:08.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:08.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:08.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:08.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:08.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:08.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:08.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:09.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15723787 2024-04-25T08:32:09.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15723787 '!=' null 2024-04-25T08:32:09.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:09.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:09.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:09.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:09.431 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:09.432 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:09.432 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:09.432 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:09.432 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:09.432 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:09.432 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:09.432 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:09.433 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:09.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:09.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:09.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:09.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:09.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:09.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:09.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:09.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:09.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:09.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:09.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:09.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:09.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:10.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:10.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:10.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:10.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:10.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:10.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:10.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:10.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:10.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:11.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=24459580 2024-04-25T08:32:11.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 24459580 '!=' null 2024-04-25T08:32:11.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:11.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:11.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:11.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:11.430 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:11.430 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:11.430 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:11.431 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:11.432 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:11.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:11.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:11.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:11.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:11.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:11.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:11.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:11.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:11.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:11.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:11.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:11.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:11.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:12.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:12.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:12.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:12.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:12.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:12.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:12.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:12.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:12.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:13.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597761 2024-04-25T08:32:13.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597761 '!=' null 2024-04-25T08:32:13.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:13.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:13.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:13.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:13.536 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:13.536 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:13.536 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:13.536 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:13.536 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:13.536 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:13.536 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:13.536 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:13.536 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:13.537 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:13.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:13.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:13.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:13.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:13.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:13.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:13.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:13.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:13.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:14.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:14.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:14.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:14.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:14.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:14.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:14.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:14.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:14.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:14.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:14.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:14.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:14.776 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:15.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=22713192 2024-04-25T08:32:15.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 22713192 '!=' null 2024-04-25T08:32:15.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:15.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:15.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:15.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:15.547 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:15.547 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:15.547 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:15.547 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:15.547 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:15.548 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:15.549 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:15.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:15.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:15.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:15.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:16.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:16.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:16.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:16.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:16.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:16.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:16.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:16.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:17.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851213 2024-04-25T08:32:17.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851213 '!=' null 2024-04-25T08:32:17.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:17.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:17.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:17.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:17.480 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:17.480 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:17.481 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:17.482 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:17.482 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:17.482 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:17.482 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:17.482 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:17.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:17.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:17.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:17.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:17.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:17.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:17.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:17.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:17.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:18.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:18.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:18.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:18.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:18.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:18.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:19.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14851397 2024-04-25T08:32:19.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14851397 '!=' null 2024-04-25T08:32:19.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:19.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:19.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:19.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:19.694 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:19.694 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:19.694 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:19.694 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:19.694 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:19.695 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:19.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:19.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:19.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:19.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:19.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:19.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:19.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:19.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:19.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:20.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:20.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:20.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:20.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:20.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:20.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:20.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:20.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:20.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:20.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:20.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:20.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:20.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:21.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=20966728 2024-04-25T08:32:21.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 20966728 '!=' null 2024-04-25T08:32:21.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:21.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:21.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:21.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:21.539 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:21.540 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:21.540 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:21.540 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:21.540 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:21.540 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:21.540 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:21.540 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:21.540 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:21.540 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:21.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:21.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:21.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:21.654 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:21.654 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:21.654 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:21.654 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:21.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:21.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:22.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:22.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:22.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:22.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:22.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:22.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:22.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:22.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:22.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:22.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:22.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:22.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:22.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:23.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12230649 2024-04-25T08:32:23.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12230649 '!=' null 2024-04-25T08:32:23.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:23.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:23.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:23.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:23.515 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:23.515 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:23.515 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:23.515 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:23.516 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:23.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:23.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:23.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:23.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:23.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:23.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:23.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:23.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:23.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:24.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:24.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:24.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:24.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:24.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:24.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:25.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472143 2024-04-25T08:32:25.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472143 '!=' null 2024-04-25T08:32:25.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:25.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:25.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:25.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:25.402 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:25.402 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:25.402 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:25.403 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:25.404 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:25.404 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:25.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:25.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:25.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:25.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:25.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:25.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:25.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:25.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:25.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:26.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:26.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:26.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:26.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:26.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:26.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:26.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:26.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:26.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:26.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:26.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:26.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:26.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:27.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356632 2024-04-25T08:32:27.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356632 '!=' null 2024-04-25T08:32:27.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:27.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:27.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:27.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:27.523 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:27.524 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:27.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:27.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:27.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:27.638 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:27.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:27.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:27.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:27.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:27.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:28.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:28.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:28.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:28.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:28.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:28.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:28.701 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:28.701 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:29.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356610 2024-04-25T08:32:29.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356610 '!=' null 2024-04-25T08:32:29.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:29.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:29.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:29.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:29.472 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:29.472 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:29.473 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:29.474 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:29.474 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:29.474 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:29.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:29.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:29.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:29.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:29.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:29.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:29.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:29.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:29.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:30.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:30.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:30.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:30.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:30.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:30.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:30.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:30.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:30.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:30.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:30.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:30.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:30.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:31.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471645 2024-04-25T08:32:31.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471645 '!=' null 2024-04-25T08:32:31.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:31.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:31.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:31.180 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:31.290 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:31.291 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:31.292 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:31.292 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:31.292 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:31.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:31.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:31.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:31.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:31.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:31.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:31.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:31.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:31.407 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:31.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:31.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:31.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:31.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:32.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:32.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:32.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:32.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:32.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:32.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:32.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:32.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:32.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:33.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356709 2024-04-25T08:32:33.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356709 '!=' null 2024-04-25T08:32:33.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:33.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:33.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:33.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:33.405 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:33.406 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:33.406 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:33.406 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:33.406 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:33.406 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:33.406 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:33.406 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:33.406 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:33.406 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:33.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:33.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:33.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:33.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:33.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:33.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:33.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:33.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:33.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:33.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:33.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:33.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:33.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:34.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:34.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:35.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598078 2024-04-25T08:32:35.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598078 '!=' null 2024-04-25T08:32:35.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:35.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:35.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:35.040 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:35.137 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:35.138 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:35.139 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:35.139 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:35.139 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:35.139 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:35.139 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:35.139 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:35.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:35.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:35.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:35.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:35.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:35.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:35.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:35.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:35.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:35.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:35.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:35.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:35.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:36.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:36.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:36.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:36.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:36.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:36.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:36.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:36.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:36.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:36.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356682 2024-04-25T08:32:36.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356682 '!=' null 2024-04-25T08:32:36.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:36.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:36.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:36.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:37.073 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:37.073 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:37.073 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:37.073 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:37.073 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:37.073 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:37.073 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:37.073 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:37.073 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:37.074 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:37.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:37.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:37.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:37.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:37.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:37.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:37.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:37.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:37.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:37.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:37.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:37.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:37.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:38.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:38.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:38.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:38.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:38.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:38.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:38.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:38.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:38.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:39.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356583 2024-04-25T08:32:39.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356583 '!=' null 2024-04-25T08:32:39.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:39.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:39.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:39.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:39.341 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:39.342 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:39.343 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:39.343 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:39.343 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:39.343 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:39.343 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:39.343 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:39.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:39.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:39.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:39.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:39.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:39.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:39.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:39.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:39.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:39.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:39.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:39.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:39.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:40.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:40.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:40.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:40.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:40.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:40.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:40.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:40.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:40.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:41.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471468 2024-04-25T08:32:41.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471468 '!=' null 2024-04-25T08:32:41.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:41.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:41.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:41.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:41.205 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:41.206 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:41.206 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:41.206 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:41.206 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:41.206 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:41.206 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:41.206 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:41.206 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:41.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:41.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:41.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:41.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:41.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:41.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:41.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:41.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:41.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:41.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:41.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:41.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:41.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:42.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:42.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:42.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:42.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:42.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:42.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:42.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:42.321 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:42.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:43.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356542 2024-04-25T08:32:43.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356542 '!=' null 2024-04-25T08:32:43.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:43.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:43.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:43.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:43.263 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:43.263 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:43.263 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:43.263 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:43.263 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:43.263 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:43.263 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:43.264 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:43.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:43.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:43.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:43.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:43.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:43.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:43.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:43.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:43.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:43.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:43.806 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:43.806 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:43.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:44.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:44.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:44.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:45.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597748 2024-04-25T08:32:45.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597748 '!=' null 2024-04-25T08:32:45.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:45.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:45.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:45.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:45.311 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:45.311 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:45.311 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:45.311 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:45.311 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:45.312 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:45.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:45.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:45.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:45.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:45.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:45.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:45.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:45.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:45.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:45.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:45.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:45.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:45.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:46.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:46.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:46.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:46.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:46.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:46.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:46.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:46.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:46.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:47.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356266 2024-04-25T08:32:47.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356266 '!=' null 2024-04-25T08:32:47.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:47.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:47.079 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:47.079 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:47.185 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:47.186 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:47.186 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:47.186 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:47.186 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:47.186 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:47.186 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:47.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:47.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:47.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:47.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:47.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:47.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:47.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:47.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:47.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:47.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:47.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:47.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:47.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:48.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:48.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:48.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:48.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:48.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:48.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:48.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:48.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:48.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:49.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356304 2024-04-25T08:32:49.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356304 '!=' null 2024-04-25T08:32:49.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:49.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:49.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:49.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:49.471 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:49.472 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:49.473 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:49.473 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:49.473 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:49.473 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:49.473 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:49.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:49.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:49.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:49.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:49.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:49.587 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:49.587 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:49.587 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:49.587 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:50.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:50.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:50.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:50.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:50.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:50.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:50.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:50.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:50.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:51.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471661 2024-04-25T08:32:51.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471661 '!=' null 2024-04-25T08:32:51.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:51.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:51.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:51.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:51.520 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:51.520 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:51.521 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:51.522 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:51.522 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:51.522 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:51.522 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:51.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:51.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:51.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:51.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:51.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:51.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:51.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:51.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:51.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:52.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:52.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:52.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:52.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:52.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:52.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:52.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:52.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:52.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:52.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:52.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:52.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:52.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:53.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356689 2024-04-25T08:32:53.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356689 '!=' null 2024-04-25T08:32:53.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:53.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:53.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:53.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:53.785 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:53.786 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:53.786 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:53.786 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:53.786 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:53.786 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:53.786 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:53.786 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:53.786 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:53.786 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:53.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:53.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:53.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:53.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:53.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:53.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:53.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:53.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:53.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:54.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:54.394 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:54.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:54.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:55.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:55.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:55.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:55.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:55.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:55.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:55.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:55.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:55.104 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:55.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471810 2024-04-25T08:32:55.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471810 '!=' null 2024-04-25T08:32:55.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:55.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:55.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:55.757 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:55.865 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:55.865 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:55.865 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:55.865 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:55.865 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:55.865 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:55.866 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:55.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:55.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:55.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:55.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:55.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:55.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:55.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:55.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:55.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:56.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:56.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:56.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:56.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:57.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:57.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:57.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12230235 2024-04-25T08:32:57.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12230235 '!=' null 2024-04-25T08:32:57.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:57.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:57.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:57.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:57.906 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:57.907 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:57.907 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:57.907 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:57.907 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:57.907 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:57.907 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:57.907 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:57.907 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:57.907 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:57.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:58.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:58.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:58.021 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:58.021 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:58.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:58.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:58.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:58.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:32:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:32:58.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:32:58.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:32:58.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:32:59.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:32:59.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:32:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:32:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:32:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:32:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:32:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:32:59.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:32:59.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:32:59.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12230558 2024-04-25T08:32:59.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12230558 '!=' null 2024-04-25T08:32:59.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:32:59.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:59.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:32:59.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:32:59.699 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:32:59.700 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:32:59.700 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:32:59.700 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:32:59.700 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:32:59.700 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:32:59.700 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:32:59.700 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:32:59.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:32:59.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:32:59.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:32:59.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:32:59.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:32:59.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:32:59.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:32:59.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:32:59.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:00.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:00.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:00.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:00.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:00.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:00.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:00.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:00.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:00.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:00.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:00.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:00.896 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:00.896 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17472208 2024-04-25T08:33:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17472208 '!=' null 2024-04-25T08:33:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:01.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:01.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:01.844 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:01.844 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:01.844 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:01.844 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:01.844 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:01.844 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:01.844 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:01.845 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:01.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:01.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:01.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:01.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:01.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:01.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:01.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:01.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:01.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:02.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:02.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:02.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:02.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:03.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:03.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:03.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:03.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11357002 2024-04-25T08:33:03.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11357002 '!=' null 2024-04-25T08:33:03.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:03.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:03.778 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:03.778 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:03.884 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:03.885 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:03.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:03.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:03.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:03.998 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:03.998 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:03.999 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:03.999 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:03.999 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:03.999 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:04.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:04.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:04.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:04.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:05.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:05.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:05.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:05.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:05.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:05.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:05.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:05.120 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:05.120 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:05.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598748 2024-04-25T08:33:05.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598748 '!=' null 2024-04-25T08:33:05.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:05.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:05.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:05.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:05.952 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:05.952 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:05.952 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:05.953 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:05.954 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:05.954 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:05.954 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:05.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:06.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:06.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:06.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:06.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:06.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:06.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:06.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:06.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:06.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:06.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:06.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:06.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:07.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:07.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:07.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:07.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:07.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:07.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:07.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:07.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:07.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10483544 2024-04-25T08:33:08.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10483544 '!=' null 2024-04-25T08:33:08.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:08.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:08.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:08.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:08.556 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:08.557 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:08.557 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:08.557 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:08.557 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:08.557 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:08.557 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:08.557 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:08.557 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:08.557 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:08.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:08.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:08.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:08.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:08.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:08.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:08.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:08.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:08.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:09.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:09.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:09.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:09.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:09.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:09.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:09.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:09.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:09.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:09.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:09.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:09.735 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:09.735 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:10.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725072 2024-04-25T08:33:10.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725072 '!=' null 2024-04-25T08:33:10.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:10.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:10.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:10.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:10.652 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:10.653 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:10.653 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:10.653 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:10.653 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:10.653 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:10.653 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:10.653 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:10.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:10.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:10.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:10.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:10.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:10.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:10.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:10.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:10.767 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:11.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:11.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:11.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:11.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:11.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:11.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:11.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:11.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:11.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:11.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:11.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:11.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:11.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:12.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=15725072 2024-04-25T08:33:12.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 15725072 '!=' null 2024-04-25T08:33:12.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:12.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:12.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:12.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:12.651 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:12.651 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:12.651 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:12.651 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:12.651 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:12.651 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:12.651 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:12.651 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:12.651 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:12.652 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:12.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:12.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:12.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:12.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:12.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:12.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:12.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:12.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:12.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:13.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:13.238 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:13.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:13.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:13.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:13.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:13.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:13.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:13.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:13.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:13.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:13.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:13.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:14.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598239 2024-04-25T08:33:14.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598239 '!=' null 2024-04-25T08:33:14.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:14.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:14.561 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:14.561 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:14.669 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:14.669 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:14.669 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:14.669 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:14.669 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:14.669 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:14.669 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:14.669 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:14.669 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:14.670 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:14.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:14.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:14.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:14.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:14.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:14.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:14.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:14.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:14.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:15.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:15.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:15.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:15.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:15.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:15.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:15.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:15.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:15.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:15.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:15.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:15.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:15.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:16.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16598239 2024-04-25T08:33:16.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16598239 '!=' null 2024-04-25T08:33:16.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:16.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:16.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:16.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:16.495 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:16.495 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:16.495 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:16.495 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:16.495 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:16.495 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:16.496 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:16.497 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:16.497 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:16.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:16.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:16.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:16.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:16.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:16.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:16.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:16.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:16.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:17.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:17.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:17.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:17.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:17.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:17.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:17.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:17.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:17.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:17.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:17.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:17.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:17.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:18.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356720 2024-04-25T08:33:18.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356720 '!=' null 2024-04-25T08:33:18.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:18.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:18.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:18.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:18.720 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:18.721 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:18.721 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:18.721 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:18.721 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:18.721 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:18.721 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:18.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:18.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:18.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:18.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:18.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:18.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:18.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:18.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:18.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:19.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:19.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:19.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:19.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:20.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:20.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:20.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:20.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:20.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:20.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:20.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:20.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:20.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:20.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356720 2024-04-25T08:33:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356720 '!=' null 2024-04-25T08:33:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:20.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:20.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:20.913 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:20.913 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:20.913 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:20.914 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:20.915 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:20.915 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:20.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:21.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:21.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:21.027 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:21.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:21.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:21.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:21.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:21.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:21.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:21.603 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:21.603 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:21.603 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:22.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:22.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:22.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:22.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:22.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:22.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:22.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:22.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:22.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:23.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356849 2024-04-25T08:33:23.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356849 '!=' null 2024-04-25T08:33:23.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:23.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:23.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:23.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:23.157 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:23.158 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:23.158 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:23.158 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:23.158 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:23.158 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:23.158 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:23.158 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:23.158 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:23.158 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:23.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:23.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:23.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:23.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:23.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:23.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:23.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:23.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:23.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:23.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:23.694 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:23.694 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:23.694 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:24.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:24.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:24.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:24.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:24.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:24.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:24.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:24.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:24.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:24.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471990 2024-04-25T08:33:24.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471990 '!=' null 2024-04-25T08:33:24.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:24.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:24.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:24.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:25.000 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:25.001 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:25.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:25.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:25.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:25.116 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:25.116 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:25.116 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:25.116 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:25.116 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:25.116 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:25.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:25.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:25.539 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:25.539 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:26.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:26.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:26.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:26.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:26.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:26.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:26.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:26.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:26.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:26.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471990 2024-04-25T08:33:26.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471990 '!=' null 2024-04-25T08:33:26.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:26.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:26.863 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:26.863 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:26.973 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:26.973 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:26.974 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:26.975 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:26.975 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:26.975 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:26.975 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:26.975 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:26.975 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:26.975 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:26.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:27.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:27.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:27.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:27.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:27.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:27.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:27.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:27.090 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:27.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:27.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:27.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:27.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:28.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:28.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:28.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:28.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:28.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:28.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:28.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:28.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:28.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:29.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356526 2024-04-25T08:33:29.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356526 '!=' null 2024-04-25T08:33:29.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:29.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:29.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:29.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:29.343 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:29.343 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:29.343 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:29.343 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:29.343 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:29.344 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:29.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:29.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:29.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:29.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:29.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:29.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:29.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:29.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:29.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:29.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:29.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:29.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:29.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:30.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:30.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:31.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=16597697 2024-04-25T08:33:31.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 16597697 '!=' null 2024-04-25T08:33:31.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:31.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:31.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:31.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:31.559 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:31.560 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:31.561 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:31.561 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:31.561 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:31.561 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:31.561 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:31.561 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:31.561 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:31.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:31.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:31.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:31.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:31.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:31.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:31.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:31.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:31.675 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:32.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:32.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:32.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:32.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:32.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:32.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:32.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:32.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:32.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:32.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:32.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:32.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:32.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:33.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356300 2024-04-25T08:33:33.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356300 '!=' null 2024-04-25T08:33:33.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:33.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:33.373 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:33.373 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:33.469 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:33.470 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:33.471 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:33.471 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:33.471 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:33.471 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:33.471 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:33.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:33.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:33.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:33.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:33.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:33.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:33.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:33.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:33.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:34.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:34.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:34.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:34.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:34.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:34.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:34.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:34.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:34.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:34.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:34.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:34.601 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:34.601 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:35.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17471260 2024-04-25T08:33:35.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17471260 '!=' null 2024-04-25T08:33:35.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:35.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:35.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:35.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:35.302 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:35.303 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:35.304 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:35.304 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:35.304 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:35.304 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:35.304 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:35.304 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:35.304 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:35.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:35.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:35.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:35.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:35.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:35.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:35.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:35.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:35.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:35.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:35.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:35.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:35.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:36.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:36.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:36.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:36.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:36.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:36.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:36.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:36.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:36.336 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:37.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11356339 2024-04-25T08:33:37.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11356339 '!=' null 2024-04-25T08:33:37.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:37.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:37.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:37.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:37.119 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:37.119 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:37.119 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:37.119 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [ 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "item": "3.0", 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "prio": 190, 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "can_preempt": true 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:37.120 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:37.121 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:37.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:37.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:37.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:37.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:37.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:37.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:37.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:37.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:37.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:37.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:37.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:37.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:37.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:38.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:38.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:38.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:38.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:38.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:38.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:38.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:38.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:38.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13627724 2024-04-25T08:33:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13627724 '!=' null 2024-04-25T08:33:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:39.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:39.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:39.288 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:39.288 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:39.288 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:39.288 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:39.288 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:39.288 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:39.288 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:39.288 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:39.289 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:39.289 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:39.289 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:39.289 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:39.289 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:39.289 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:39.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:39.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:39.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:39.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:39.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:39.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:39.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:39.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:39.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:39.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:39.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:39.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:39.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:40.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:40.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:40.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:40.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:40.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:40.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:40.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:40.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:40.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:40.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=17034695 2024-04-25T08:33:40.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 17034695 '!=' null 2024-04-25T08:33:40.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:40.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:40.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:40.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:41.042 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:41.042 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:41.042 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:41.042 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:41.042 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:41.042 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:41.042 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:41.042 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:41.042 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:41.043 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:41.043 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:41.043 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:41.043 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:41.043 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:41.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:41.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:41.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:41.156 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:41.156 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:41.156 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:41.156 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:41.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:41.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:41.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:41.557 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:41.557 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:41.557 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:42.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:42.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:42.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:42.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:42.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:42.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:42.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:42.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:42.102 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:42.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7862050 2024-04-25T08:33:42.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7862050 '!=' null 2024-04-25T08:33:42.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:42.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:42.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:42.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:42.759 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:42.759 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:42.759 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:42.759 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:42.759 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:42.759 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:42.759 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:42.759 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:42.760 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:42.760 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:42.760 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:42.760 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:42.760 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:42.760 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:42.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:42.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:42.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:42.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:42.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:42.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:42.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:42.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:42.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:43.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T08:33:43.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:43.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:43.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:43.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 3 2024-04-25T08:33:43.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T08:33:43.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:33:43.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:33:43.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:33:43.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:33:43.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:33:43.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:33:43.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:33:44.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7862238 2024-04-25T08:33:44.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7862238 '!=' null 2024-04-25T08:33:44.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:33:44.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:44.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: CEPH_ARGS= 2024-04-25T08:33:44.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_recovery_reservations 2024-04-25T08:33:44.532 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "local_reservations": { 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "remote_reservations": { 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "max_allowed": 1, 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "min_priority": 0, 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "queues": [], 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: "in_progress": [] 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:44.533 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:44.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:33:44.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:33:44.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:33:44.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:33:44.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:33:44.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:33:44.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:33:44.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:33:44.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:33:45.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=3 2024-04-25T08:33:45.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:33:45.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:33:45.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:33:45.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 3 = 3 2024-04-25T08:33:45.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:33:45.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:33:45.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:298: TEST_recovery_priority: ceph pg dump pgs 2024-04-25T08:33:45.974 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:33:45.974 INFO:tasks.workunit.client.0.smithi134.stdout:7.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:28:57.848053+0000 87'600 114:1140 [1,4] 1 [1,4] 1 0'0 2024-04-25T08:17:37.191617+0000 0'0 2024-04-25T08:17:37.191617+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:38:09.320978+0000 0 0 2024-04-25T08:33:45.974 INFO:tasks.workunit.client.0.smithi134.stdout:3.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:33:38.857081+0000 87'600 114:1175 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:17:23.969579+0000 0'0 2024-04-25T08:17:23.969579+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:30:50.296503+0000 0 0 2024-04-25T08:33:45.975 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 200 0 0 0 0 2097152000 0 0 600 0 600 active+clean 2024-04-25T08:31:27.727533+0000 87'600 114:1188 [1,2] 1 [1,2] 1 0'0 2024-04-25T08:17:17.811742+0000 0'0 2024-04-25T08:17:17.811742+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:10:12.008225+0000 0 0 2024-04-25T08:33:45.975 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:33:45.975 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:33:45.975 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:33:45.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: get_asok_path osd.1 2024-04-25T08:33:45.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T08:33:45.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T08:33:45.992 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:33:45.992 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:45.992 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:33:45.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.206040/ceph-osd.1.asok 2024-04-25T08:33:45.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: CEPH_ARGS= 2024-04-25T08:33:45.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:300: TEST_recovery_priority: ceph --admin-daemon /tmp/ceph-asok.206040/ceph-osd.1.asok dump_pgstate_history 2024-04-25T08:33:46.103 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T08:33:46.103 INFO:tasks.workunit.client.0.smithi134.stdout: "pgs": [ 2024-04-25T08:33:46.103 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.103 INFO:tasks.workunit.client.0.smithi134.stdout: "pg": "7.0", 2024-04-25T08:33:46.103 INFO:tasks.workunit.client.0.smithi134.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.103 INFO:tasks.workunit.client.0.smithi134.stdout: "history": [ 2024-04-25T08:33:46.103 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.103 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "58", 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:37.978954+0000", 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:37.978973+0000" 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:37.978986+0000", 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:38.144886+0000" 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.104 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:38.144888+0000", 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:38.145029+0000" 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:38.145030+0000", 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:38.145055+0000" 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:38.145056+0000", 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:38.778706+0000" 2024-04-25T08:33:46.105 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:37.978980+0000", 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:38.778717+0000" 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:38.779014+0000", 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.111255+0000" 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.106 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:38.778718+0000", 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.111273+0000" 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:37.978974+0000", 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.111294+0000" 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:37.978947+0000", 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.111299+0000" 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.107 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "58", 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.111299+0000", 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.111500+0000" 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "85", 2024-04-25T08:33:46.108 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.111503+0000", 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.111512+0000" 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.111518+0000", 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.264620+0000" 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.264621+0000", 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.264747+0000" 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.109 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.264748+0000", 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.264769+0000" 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.264770+0000", 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.765627+0000" 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.111515+0000", 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:40.765646+0000" 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.110 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.765995+0000", 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:41.603596+0000" 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:41.603598+0000", 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:41.603820+0000" 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:41.603821+0000", 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.843841+0000" 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.111 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.765649+0000", 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.843882+0000" 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.111513+0000", 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.843948+0000" 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:40.111500+0000", 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.843955+0000" 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.112 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "85", 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.843956+0000", 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.844321+0000" 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "92", 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.113 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.844332+0000", 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.844356+0000" 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.844371+0000", 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:13.033329+0000" 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:13.033330+0000", 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:13.033411+0000" 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:13.033411+0000", 2024-04-25T08:33:46.114 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:13.033427+0000" 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:13.033427+0000", 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:13.778520+0000" 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.844364+0000", 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:13.778529+0000" 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.115 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:13.778609+0000", 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:13.977397+0000" 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:13.977399+0000", 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:13.977479+0000" 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:13.977480+0000", 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:52.991097+0000" 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.116 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:13.778532+0000", 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:52.991140+0000" 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.844357+0000", 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:52.991335+0000" 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.844322+0000", 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:52.991344+0000" 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.117 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "92", 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:52.991345+0000", 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:52.992045+0000" 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "94", 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:52.992058+0000", 2024-04-25T08:33:46.118 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:52.992090+0000" 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:52.992109+0000", 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.133970+0000" 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.133972+0000", 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.134234+0000" 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:52.992100+0000", 2024-04-25T08:33:46.119 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.134277+0000" 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.134278+0000", 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.907308+0000" 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:52.992091+0000", 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.907390+0000" 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:52.992046+0000", 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.907400+0000" 2024-04-25T08:33:46.120 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "94", 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.907402+0000", 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.907898+0000" 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "109", 2024-04-25T08:33:46.121 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.907911+0000", 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.907945+0000" 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.907966+0000", 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.909053+0000" 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.909054+0000", 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.909216+0000" 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.122 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.909217+0000", 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:53.909252+0000" 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.909253+0000", 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:54.530968+0000" 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.907957+0000", 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:54.531015+0000" 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.123 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:54.535232+0000", 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:55.492612+0000" 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:55.492613+0000", 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:55.492930+0000" 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:55.492931+0000", 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:25:55.493553+0000" 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:55.493555+0000", 2024-04-25T08:33:46.124 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:27.698853+0000" 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:27.698855+0000", 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:27.699234+0000" 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:27.699234+0000", 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:40.733965+0000" 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:33:46.125 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:40.733966+0000", 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:40.734518+0000" 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:40.734518+0000", 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:28:54.918338+0000" 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:28:54.918339+0000", 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:28:55.930713+0000" 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.126 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:54.531016+0000", 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:28:55.930759+0000" 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.907946+0000", 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:28:55.930798+0000" 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:25:53.907899+0000", 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:28:55.930806+0000" 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.127 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "109", 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:28:55.930807+0000", 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:28:55.931303+0000" 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: "pg": "3.0", 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: "history": [ 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "37", 2024-04-25T08:33:46.128 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:24.930287+0000", 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:24.930325+0000" 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:24.930350+0000", 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:25.160684+0000" 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:25.160689+0000", 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:25.160919+0000" 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.129 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:25.160921+0000", 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:25.160967+0000" 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:25.160968+0000", 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:25.561608+0000" 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:24.930338+0000", 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:25.561620+0000" 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.130 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:25.561900+0000", 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:25.863896+0000" 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:25.863898+0000", 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:25.864047+0000" 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:25.864048+0000", 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:26.774406+0000" 2024-04-25T08:33:46.131 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:25.561621+0000", 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:26.774442+0000" 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:24.930327+0000", 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:26.774528+0000" 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:24.930276+0000", 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:26.774536+0000" 2024-04-25T08:33:46.132 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "37", 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:26.774537+0000", 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:26.774942+0000" 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "83", 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.133 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:26.774951+0000", 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:26.774977+0000" 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:26.774993+0000", 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:26.943141+0000" 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:26.943143+0000", 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:26.943296+0000" 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.134 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:26.943297+0000", 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:26.943330+0000" 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:26.943331+0000", 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:27.645534+0000" 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:26.774985+0000", 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:27.645574+0000" 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.135 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:27.646078+0000", 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:28.575826+0000" 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:28.575827+0000", 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:28.575919+0000" 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:28.575919+0000", 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.841243+0000" 2024-04-25T08:33:46.136 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:27.645575+0000", 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.841278+0000" 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:26.774977+0000", 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.841354+0000" 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:26.774943+0000", 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.841361+0000" 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.137 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "85", 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.841362+0000", 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.841858+0000" 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "100", 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.138 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.841868+0000", 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.841896+0000" 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.841914+0000", 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.842167+0000" 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.842169+0000", 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.842308+0000" 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.139 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.842309+0000", 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:12.842337+0000" 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.842337+0000", 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:13.778753+0000" 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.841906+0000", 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:13.778761+0000" 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.140 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:13.778822+0000", 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:14.098688+0000" 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:14.098690+0000", 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:18:14.098773+0000" 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:14.098774+0000", 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:12.287502+0000" 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:13.778761+0000", 2024-04-25T08:33:46.141 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:12.287569+0000" 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.841897+0000", 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:12.287912+0000" 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:18:12.841858+0000", 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:12.287921+0000" 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.142 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "101", 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:12.287922+0000", 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:12.288678+0000" 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "103", 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:12.288691+0000", 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:12.288722+0000" 2024-04-25T08:33:46.143 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:12.288741+0000", 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:12.429462+0000" 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:12.429464+0000", 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:12.429612+0000" 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:12.288732+0000", 2024-04-25T08:33:46.144 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:12.429633+0000" 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:12.429634+0000", 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.092167+0000" 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:12.288723+0000", 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.092233+0000" 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:12.288678+0000", 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.092246+0000" 2024-04-25T08:33:46.145 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "103", 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.092247+0000", 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.092740+0000" 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "113", 2024-04-25T08:33:46.146 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.092753+0000", 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.092788+0000" 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.092811+0000", 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.232443+0000" 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.232444+0000", 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.232615+0000" 2024-04-25T08:33:46.147 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.232616+0000", 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.232652+0000" 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.232652+0000", 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.776494+0000" 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.092799+0000", 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:13.776519+0000" 2024-04-25T08:33:46.148 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.779461+0000", 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:14.730523+0000" 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:14.730525+0000", 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:27.699117+0000" 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:27.699119+0000", 2024-04-25T08:33:46.149 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:27.699582+0000" 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:27.699585+0000", 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:40.733858+0000" 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/NotRecovering", 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:40.733858+0000", 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:40.734040+0000" 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:33:46.150 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:40.734040+0000", 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:31:24.640603+0000" 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:31:24.640604+0000", 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:31:24.641128+0000" 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:31:24.641128+0000", 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:33:37.768587+0000" 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:33:37.768588+0000", 2024-04-25T08:33:46.151 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:33:38.249697+0000" 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.776520+0000", 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:33:38.249725+0000" 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.092789+0000", 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:33:38.249759+0000" 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:13.092740+0000", 2024-04-25T08:33:46.152 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:33:38.249767+0000" 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "113", 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:33:38.249768+0000", 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:33:38.250659+0000" 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.153 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: "pg": "1.0", 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: "currently": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: "history": [ 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "25", 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Initial", 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.067795+0000", 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:18.067935+0000" 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.154 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "25", 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.067936+0000", 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:18.067975+0000" 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "27", 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.155 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.067978+0000", 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:18.067988+0000" 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.067996+0000", 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:18.214363+0000" 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.214364+0000", 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:18.214513+0000" 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.156 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.214514+0000", 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:18.214540+0000" 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.214540+0000", 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:18.581750+0000" 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.067992+0000", 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:18.581778+0000" 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.582320+0000", 2024-04-25T08:33:46.157 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.370173+0000" 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.370174+0000", 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.370274+0000" 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.370274+0000", 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.946522+0000" 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.581798+0000", 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.946565+0000" 2024-04-25T08:33:46.158 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.067989+0000", 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.946652+0000" 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:18.067975+0000", 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.946660+0000" 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "27", 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.159 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.946661+0000", 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.947100+0000" 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "96", 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.160 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.947125+0000", 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.947150+0000" 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.947167+0000", 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.947878+0000" 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.947880+0000", 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.948149+0000" 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.161 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.948151+0000", 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:19.948205+0000" 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.948206+0000", 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:20.783295+0000" 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.947159+0000", 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:20.783333+0000" 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.162 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:20.783844+0000", 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:21.386696+0000" 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:21.386697+0000", 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:17:21.386807+0000" 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Clean", 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:21.386808+0000", 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.005535+0000" 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.163 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:20.783335+0000", 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.005588+0000" 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.947151+0000", 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.005782+0000" 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:17:19.947101+0000", 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.005791+0000" 2024-04-25T08:33:46.164 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "96", 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.005791+0000", 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.006510+0000" 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.165 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "98", 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.006521+0000", 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.006551+0000" 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.006569+0000", 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.181725+0000" 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.166 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.181727+0000", 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.181898+0000" 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.006561+0000", 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.181919+0000" 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/WaitActingChange", 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.181920+0000", 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.985186+0000" 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.167 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.006552+0000", 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.985243+0000" 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.006511+0000", 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.985259+0000" 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "98", 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.168 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.985260+0000", 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.985716+0000" 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "111", 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Start", 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.985727+0000", 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:04.985756+0000" 2024-04-25T08:33:46.169 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetInfo", 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.985775+0000", 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:05.195353+0000" 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetLog", 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:05.195355+0000", 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:05.195550+0000" 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/GetMissing", 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:05.195551+0000", 2024-04-25T08:33:46.170 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:05.195589+0000" 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:05.195590+0000", 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:05.655060+0000" 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Peering", 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.985766+0000", 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:05.655085+0000" 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Activating", 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:05.658216+0000", 2024-04-25T08:33:46.171 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:26:06.228211+0000" 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitLocalRecoveryReserved", 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:06.228213+0000", 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:28:54.918674+0000" 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/WaitRemoteRecoveryReserved", 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:28:54.918675+0000", 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:28:54.919433+0000" 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovering", 2024-04-25T08:33:46.172 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:28:54.919435+0000", 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:31:24.640390+0000" 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active/Recovered", 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:31:24.640391+0000", 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:31:25.311843+0000" 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary/Active", 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:05.655086+0000", 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:31:25.311892+0000" 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.173 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started/Primary", 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.985757+0000", 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:31:25.311940+0000" 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Started", 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:26:04.985717+0000", 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:31:25.311949+0000" 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": "111", 2024-04-25T08:33:46.174 INFO:tasks.workunit.client.0.smithi134.stdout: "states": [ 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "Reset", 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: "enter": "2024-04-25T08:31:25.311950+0000", 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: "exit": "2024-04-25T08:31:25.312550+0000" 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T08:33:46.175 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T08:33:46.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:302: TEST_recovery_priority: '[' 0 '!=' 0 ']' 2024-04-25T08:33:46.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:306: TEST_recovery_priority: echo TEST PASSED 2024-04-25T08:33:46.176 INFO:tasks.workunit.client.0.smithi134.stdout:TEST PASSED 2024-04-25T08:33:46.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:309: TEST_recovery_priority: delete_pool test1 2024-04-25T08:33:46.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:33:46.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:33:46.689 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test1' does not exist 2024-04-25T08:33:46.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:310: TEST_recovery_priority: delete_pool test3 2024-04-25T08:33:46.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test3 2024-04-25T08:33:46.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2024-04-25T08:33:47.538 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test3' does not exist 2024-04-25T08:33:47.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:311: TEST_recovery_priority: delete_pool test7 2024-04-25T08:33:47.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test7 2024-04-25T08:33:47.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2024-04-25T08:33:48.304 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test7' does not exist 2024-04-25T08:33:48.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:312: TEST_recovery_priority: kill_daemons td/osd-recovery-prio 2024-04-25T08:33:48.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:33:48.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:33:48.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:33:48.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:33:48.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:33:49.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:33:49.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:313: TEST_recovery_priority: return 0 2024-04-25T08:33:49.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-prio.sh:41: run: teardown td/osd-recovery-prio 2024-04-25T08:33:49.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:33:49.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:33:49.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:33:49.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:33:49.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:33:49.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:33:49.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:33:49.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:33:49.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:33:49.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:33:49.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:33:49.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:33:49.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:33:49.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:33:49.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:33:49.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:49.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:33:49.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:33:49.652 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:49.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:33:49.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:33:49.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:33:49.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:33:49.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:49.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:33:49.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.206040 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-prio 0 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-prio 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-prio KILL 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:33:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:33:49.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:33:49.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:33:49.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:33:49.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:33:49.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:33:49.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:33:49.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:33:49.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:33:49.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:49.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:33:49.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:33:49.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:49.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:33:49.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:33:49.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-prio 2024-04-25T08:33:49.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:33:49.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:49.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.206040 2024-04-25T08:33:49.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.206040 2024-04-25T08:33:49.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:33:49.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:33:49.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:33:49.752 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:33:49.753 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:33:49.764 INFO:tasks.workunit:Running workunit osd/osd-recovery-space.sh... 2024-04-25T08:33:49.764 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-recovery-space.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh 2024-04-25T08:33:49.819 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:33:49.825 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:33:49.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:33:49.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:33:49.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:33:49.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:33:49.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-space 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:21: run: local dir=td/osd-recovery-space 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:22: run: shift 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: export CEPH_MON=127.0.0.1:7221 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:24: run: CEPH_MON=127.0.0.1:7221 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:25: run: export CEPH_ARGS 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: uuidgen 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:26: run: CEPH_ARGS+='--fsid=59c4ff4d-62f3-43b9-8fb7-3fa16eda36eb --auth-supported=none ' 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7221 ' 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:28: run: CEPH_ARGS+='--osd_max_backfills=10 ' 2024-04-25T08:33:49.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:29: run: CEPH_ARGS+='--osd_mclock_override_recovery_settings=true ' 2024-04-25T08:33:49.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: export objects=600 2024-04-25T08:33:49.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:30: run: objects=600 2024-04-25T08:33:49.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: export poolprefix=test 2024-04-25T08:33:49.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:31: run: poolprefix=test 2024-04-25T08:33:49.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: set 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:33: run: local funcs=TEST_recovery_test_simple 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:34: run: for func in $funcs 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:35: run: setup td/osd-recovery-space 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-space 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-space 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:33:49.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:33:49.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:33:49.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:33:49.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:33:49.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:33:49.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:33:49.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:33:49.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:33:49.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:49.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:33:49.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:33:49.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:33:49.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:33:49.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:33:49.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:33:49.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:33:49.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:49.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:33:49.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.263450 2024-04-25T08:33:49.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:33:49.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:33:49.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-space 2024-04-25T08:33:49.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:33:49.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:49.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:33:49.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.263450 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-space 1' TERM HUP INT 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:36: run: TEST_recovery_test_simple td/osd-recovery-space 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:89: TEST_recovery_test_simple: local dir=td/osd-recovery-space 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:90: TEST_recovery_test_simple: local pools=1 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:91: TEST_recovery_test_simple: local OSDS=2 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:93: TEST_recovery_test_simple: run_mon td/osd-recovery-space a 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-space 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:33:49.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:33:49.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:33:49.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-space/a 2024-04-25T08:33:49.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-space/a --run-dir=td/osd-recovery-space 2024-04-25T08:33:50.337 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:33:50.337 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:33:50.337 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:33:50.338 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:33:50.338 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:50.338 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:33:50.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.263450/$cluster-$name.asok' 2024-04-25T08:33:50.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-space/a '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.263450/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-space/log --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:33:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:33:50.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:33:50.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:33:50.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:33:50.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.263450/ceph-mon.a.asok 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:33:50.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.263450/ceph-mon.a.asok config get fsid 2024-04-25T08:33:50.699 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:33:50.699 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:33:50.699 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:33:50.699 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:33:50.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:33:50.700 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:33:50.700 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:33:50.700 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:33:50.701 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:33:50.701 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:50.701 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:33:50.701 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.263450/ceph-mon.a.asok 2024-04-25T08:33:50.701 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:33:50.701 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.263450/ceph-mon.a.asok config get mon_host 2024-04-25T08:33:50.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:94: TEST_recovery_test_simple: run_mgr td/osd-recovery-space x 2024-04-25T08:33:50.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-space 2024-04-25T08:33:50.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:33:50.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:33:50.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:33:50.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-space/x 2024-04-25T08:33:50.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:33:51.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:33:51.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:33:51.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:33:51.750 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:33:51.751 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:51.751 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:33:51.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.263450/$cluster-$name.asok' 2024-04-25T08:33:51.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:33:51.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-space/x '--log-file=td/osd-recovery-space/$name.log' '--admin-socket=/tmp/ceph-asok.263450/$cluster-$name.asok' --run-dir=td/osd-recovery-space '--pid-file=td/osd-recovery-space/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:33:51.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:95: TEST_recovery_test_simple: export CEPH_ARGS 2024-04-25T08:33:51.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: expr 2 - 1 2024-04-25T08:33:51.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: seq 0 1 2024-04-25T08:33:51.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 0 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/0 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59c4ff4d-62f3-43b9-8fb7-3fa16eda36eb --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/0' 2024-04-25T08:33:51.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/0/journal' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.263450/$cluster-$name.asok' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.263450/$cluster-$name.asok' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:33:51.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-04-25T08:33:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-04-25T08:33:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:33:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:33:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:33:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:33:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:33:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:33:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/0 2024-04-25T08:33:51.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:33:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=acc6d3e3-16ab-4a7c-923a-b1d0bd3e128f 2024-04-25T08:33:51.827 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 acc6d3e3-16ab-4a7c-923a-b1d0bd3e128f 2024-04-25T08:33:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 acc6d3e3-16ab-4a7c-923a-b1d0bd3e128f' 2024-04-25T08:33:51.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:33:51.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvFSpm9ivLMhAAV8VoCbFTOrJ6UU+JaLdKpQ== 2024-04-25T08:33:51.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvFSpm9ivLMhAAV8VoCbFTOrJ6UU+JaLdKpQ=="}' 2024-04-25T08:33:51.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new acc6d3e3-16ab-4a7c-923a-b1d0bd3e128f -i td/osd-recovery-space/0/new.json 2024-04-25T08:33:52.168 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:33:52.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/0/new.json 2024-04-25T08:33:52.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=59c4ff4d-62f3-43b9-8fb7-3fa16eda36eb --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.263450/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBvFSpm9ivLMhAAV8VoCbFTOrJ6UU+JaLdKpQ== --osd-uuid acc6d3e3-16ab-4a7c-923a-b1d0bd3e128f 2024-04-25T08:33:52.215 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:52.218+0000 7f1e1edc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:52.245 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:52.246+0000 7f1e1edc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:52.246 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:52.250+0000 7f1e1edc27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:52.246 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:52.250+0000 7f1e1edc27c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:33:52.246 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:52.250+0000 7f1e1edc27c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:33:52.246 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:52.250+0000 7f1e1edc27c0 -1 bluestore(td/osd-recovery-space/0/block) _read_bdev_label failed to open td/osd-recovery-space/0/block: (2) No such file or directory 2024-04-25T08:33:52.285 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:52.286+0000 7f1e1edc27c0 -1 bluestore(td/osd-recovery-space/0) _read_fsid unparsable uuid 2024-04-25T08:33:58.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/0/keyring 2024-04-25T08:33:58.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:33:58.773 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:33:58.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:33:58.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:33:59.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:33:59.319 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:33:59.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=59c4ff4d-62f3-43b9-8fb7-3fa16eda36eb --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/0 --osd-journal=td/osd-recovery-space/0/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.263450/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:33:59.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:33:59.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:33:59.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:33:59.347 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:59.353+0000 7ff9862617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:59.349 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:59.357+0000 7ff9862617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:59.350 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:59.357+0000 7ff9862617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:33:59.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:33:59.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:33:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:33:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:33:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:33:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:33:59.779 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:33:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:33:59.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:33:59.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:33:59.912 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:33:59.917+0000 7ff9862617c0 -1 Falling back to public interface 2024-04-25T08:34:00.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:01.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:01.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:01.198 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:34:01.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:34:01.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:01.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:01.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:02.576 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:02.581+0000 7ff9862617c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:34:02.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:02.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:02.606 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:34:02.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:34:02.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:02.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:03.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:04.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:04.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:04.267 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:34:04.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:34:04.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:04.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:04.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:05.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:05.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:05.745 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:34:05.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:34:05.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:05.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:06.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:07.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:07.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:07.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:34:07.215 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:34:07.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:07.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:07.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:08.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:08.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:08.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:34:08.687 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:34:08.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:08.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:09.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:10.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:10.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:10.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:34:10.159 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:34:10.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:10.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:10.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:11.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:11.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:11.682 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:34:11.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:34:11.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:11.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:12.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:13.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:13.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:13.111 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:34:13.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:34:13.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:13.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:13.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:14.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:14.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:14.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:34:14.567 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:34:14.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:14.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:15.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:16.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:16.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:16.060 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:34:16.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:34:16.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:16.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:16.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:17.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:17.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:17.601 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:34:17.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:34:17.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:17.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:18.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:19.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:19.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:19.367 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:34:19.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:34:19.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:19.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:19.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:20.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:20.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:20.964 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:34:20.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:34:20.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:20.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:21.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:22.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:22.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:22.979 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:34:22.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:34:22.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:22.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:24.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:25.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:25.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:25.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:34:25.191 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:34:25.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:25.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:26.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:27.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:27.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:27.351 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:34:27.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:34:27.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:27.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:27.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:28.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:28.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:28.881 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:34:28.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:34:28.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:28.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:34:29.307 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2729116330,v1:127.0.0.1:6803/2729116330] [v2:127.0.0.1:6804/2729116330,v1:127.0.0.1:6805/2729116330] exists,up acc6d3e3-16ab-4a7c-923a-b1d0bd3e128f 2024-04-25T08:34:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:34:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:34:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:34:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:97: TEST_recovery_test_simple: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:34:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:99: TEST_recovery_test_simple: run_osd td/osd-recovery-space 1 2024-04-25T08:34:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-space 2024-04-25T08:34:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:34:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:34:29.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:34:29.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-space/1 2024-04-25T08:34:29.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59c4ff4d-62f3-43b9-8fb7-3fa16eda36eb --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true ' 2024-04-25T08:34:29.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:34:29.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-space/1' 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-space/1/journal' 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-space' 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.263450/$cluster-$name.asok' 2024-04-25T08:34:29.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.263450/$cluster-$name.asok' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-space/$name.log' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-space/$name.pid' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:34:29.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-space/1 2024-04-25T08:34:29.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:34:29.316 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 14f8d4a1-0309-4ea3-a85b-fd572c7c97e2 2024-04-25T08:34:29.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14f8d4a1-0309-4ea3-a85b-fd572c7c97e2 2024-04-25T08:34:29.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 14f8d4a1-0309-4ea3-a85b-fd572c7c97e2' 2024-04-25T08:34:29.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:34:29.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVFSpmJCB0FBAAZrn5OOnRnFTEorjJYb0bTQ== 2024-04-25T08:34:29.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVFSpmJCB0FBAAZrn5OOnRnFTEorjJYb0bTQ=="}' 2024-04-25T08:34:29.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14f8d4a1-0309-4ea3-a85b-fd572c7c97e2 -i td/osd-recovery-space/1/new.json 2024-04-25T08:34:29.827 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:34:29.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-space/1/new.json 2024-04-25T08:34:29.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=59c4ff4d-62f3-43b9-8fb7-3fa16eda36eb --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.263450/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVFSpmJCB0FBAAZrn5OOnRnFTEorjJYb0bTQ== --osd-uuid 14f8d4a1-0309-4ea3-a85b-fd572c7c97e2 2024-04-25T08:34:29.881 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:29.889+0000 7feb513be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:29.922 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:29.929+0000 7feb513be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:29.924 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:29.929+0000 7feb513be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:29.929+0000 7feb513be7c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:34:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:29.929+0000 7feb513be7c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:34:29.925 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:29.929+0000 7feb513be7c0 -1 bluestore(td/osd-recovery-space/1/block) _read_bdev_label failed to open td/osd-recovery-space/1/block: (2) No such file or directory 2024-04-25T08:34:29.953 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:29.961+0000 7feb513be7c0 -1 bluestore(td/osd-recovery-space/1) _read_fsid unparsable uuid 2024-04-25T08:34:36.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-space/1/keyring 2024-04-25T08:34:36.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:34:36.953 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:34:36.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:34:36.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-space/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:34:37.529 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:34:37.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:34:37.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=59c4ff4d-62f3-43b9-8fb7-3fa16eda36eb --auth-supported=none --mon-host=127.0.0.1:7221 --osd_max_backfills=10 --osd_mclock_override_recovery_settings=true --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-space/1 --osd-journal=td/osd-recovery-space/1/journal --chdir= --run-dir=td/osd-recovery-space '--admin-socket=/tmp/ceph-asok.263450/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-space/$name.log' '--pid-file=td/osd-recovery-space/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:34:37.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:34:37.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:34:37.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:34:37.557 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:37.560+0000 7fccac7a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:37.559 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:37.564+0000 7fccac7a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:37.560 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:37.564+0000 7fccac7a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:34:37.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:34:37.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:34:37.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:34:37.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:34:37.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:34:37.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:37.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:34:37.948 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:34:37.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:37.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:38.127 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:38.132+0000 7fccac7a17c0 -1 Falling back to public interface 2024-04-25T08:34:38.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:39.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:39.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:39.371 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:34:39.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:34:39.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:39.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:39.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:40.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:40.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:40.786 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:34:40.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:34:40.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:40.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:40.883 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:34:40.888+0000 7fccac7a17c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:34:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:42.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:42.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:42.339 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:34:42.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:34:42.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:42.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:42.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:43.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:43.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:43.939 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:34:43.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:34:43.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:43.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:44.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:45.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:45.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:45.409 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:34:45.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:34:45.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:45.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:45.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:46.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:46.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:46.943 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:34:46.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:34:46.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:46.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:47.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:48.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:48.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:48.361 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:34:48.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:34:48.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:48.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:48.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:49.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:49.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:49.910 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:34:49.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:34:49.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:49.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:50.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:51.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:51.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:51.381 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:34:51.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:34:51.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:51.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:51.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:52.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:52.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:52.943 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:34:52.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:34:52.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:52.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:53.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:54.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:54.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:54.528 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:34:54.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:34:54.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:54.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:54.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:55.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:55.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:55.999 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:34:55.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:34:55.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:56.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:56.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:57.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:57.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:57.531 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:34:57.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:34:57.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:57.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:34:58.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:34:59.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:34:59.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:34:59.778 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:34:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:34:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:34:59.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:35:00.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:01.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:01.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:01.482 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:35:01.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:35:01.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:01.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:35:02.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:03.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:03.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:03.242 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:35:03.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:35:03.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:03.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:35:04.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:05.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:05.048 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:35:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:35:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:35:06.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:07.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:07.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:07.228 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:35:07.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:35:07.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:07.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:35:07.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:35:08.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:35:08.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:35:08.775 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T08:35:08.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:35:08.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:35:08.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:35:09.241 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2205332068,v1:127.0.0.1:6811/2205332068] [v2:127.0.0.1:6812/2205332068,v1:127.0.0.1:6813/2205332068] exists,up 14f8d4a1-0309-4ea3-a85b-fd572c7c97e2 2024-04-25T08:35:09.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:35:09.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:35:09.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:35:09.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:102: TEST_recovery_test_simple: ceph osd set-nearfull-ratio .40 2024-04-25T08:35:10.151 INFO:tasks.workunit.client.0.smithi134.stderr:osd set-nearfull-ratio 0.4 2024-04-25T08:35:10.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:103: TEST_recovery_test_simple: ceph osd set-backfillfull-ratio .45 2024-04-25T08:35:11.013 INFO:tasks.workunit.client.0.smithi134.stderr:osd set-backfillfull-ratio 0.45 2024-04-25T08:35:11.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:104: TEST_recovery_test_simple: ceph osd set-full-ratio .50 2024-04-25T08:35:11.708 INFO:tasks.workunit.client.0.smithi134.stderr:osd set-full-ratio 0.5 2024-04-25T08:35:11.726 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:35:11.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:106: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-04-25T08:35:11.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:108: TEST_recovery_test_simple: create_pool test1 1 1 2024-04-25T08:35:11.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test1 1 1 2024-04-25T08:35:12.427 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test1' already exists 2024-04-25T08:35:12.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:35:13.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:109: TEST_recovery_test_simple: ceph osd pool set test1 size 1 --yes-i-really-mean-it 2024-04-25T08:35:14.226 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 1 2024-04-25T08:35:14.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:112: TEST_recovery_test_simple: wait_for_clean 2024-04-25T08:35:14.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:35:14.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:35:14.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:35:14.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:35:14.254 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:35:14.254 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:35:14.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:35:14.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:35:14.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:35:14.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:35:14.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:35:14.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:35:14.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:35:14.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:35:14.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:35:14.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:35:14.844 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T08:35:14.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:35:14.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:35:14.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:35:15.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836494 2024-04-25T08:35:15.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836494 2024-04-25T08:35:15.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494' 2024-04-25T08:35:15.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:35:15.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:35:15.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672969 2024-04-25T08:35:15.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672969 2024-04-25T08:35:15.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494 1-42949672969' 2024-04-25T08:35:15.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:35:15.162 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836494 2024-04-25T08:35:15.162 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:35:15.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:35:15.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836494 2024-04-25T08:35:15.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:35:15.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836494 2024-04-25T08:35:15.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-04-25T08:35:15.165 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836494 2024-04-25T08:35:15.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:35:15.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836492 -lt 21474836494 2024-04-25T08:35:15.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:35:16.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:35:16.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:35:17.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836494 -lt 21474836494 2024-04-25T08:35:17.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:35:17.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:35:17.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672969 2024-04-25T08:35:17.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:35:17.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672969 2024-04-25T08:35:17.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:35:17.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672969 2024-04-25T08:35:17.022 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672969 2024-04-25T08:35:17.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2024-04-25T08:35:17.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:35:17.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672969 -lt 42949672969 2024-04-25T08:35:17.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:35:17.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:35:17.455 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:35:18.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:35:18.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:35:18.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:35:18.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:35:18.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:35:18.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:35:18.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:35:18.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:35:18.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:35:18.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:35:18.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:35:18.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:35:18.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:35:18.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:35:18.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:35:18.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:114: TEST_recovery_test_simple: dd if=/dev/urandom of=td/osd-recovery-space/datafile bs=1024 count=5 2024-04-25T08:35:18.947 INFO:tasks.workunit.client.0.smithi134.stderr:5+0 records in 2024-04-25T08:35:18.947 INFO:tasks.workunit.client.0.smithi134.stderr:5+0 records out 2024-04-25T08:35:18.947 INFO:tasks.workunit.client.0.smithi134.stderr:5120 bytes (5.1 kB, 5.0 KiB) copied, 0.000156979 s, 32.6 MB/s 2024-04-25T08:35:18.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: seq 1 600 2024-04-25T08:35:18.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:18.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj1 td/osd-recovery-space/datafile 2024-04-25T08:35:19.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:19.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj2 td/osd-recovery-space/datafile 2024-04-25T08:35:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj3 td/osd-recovery-space/datafile 2024-04-25T08:35:19.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:19.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj4 td/osd-recovery-space/datafile 2024-04-25T08:35:19.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:19.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj5 td/osd-recovery-space/datafile 2024-04-25T08:35:19.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:19.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj6 td/osd-recovery-space/datafile 2024-04-25T08:35:19.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:19.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj7 td/osd-recovery-space/datafile 2024-04-25T08:35:19.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:19.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj8 td/osd-recovery-space/datafile 2024-04-25T08:35:19.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:19.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj9 td/osd-recovery-space/datafile 2024-04-25T08:35:20.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj10 td/osd-recovery-space/datafile 2024-04-25T08:35:20.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj11 td/osd-recovery-space/datafile 2024-04-25T08:35:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj12 td/osd-recovery-space/datafile 2024-04-25T08:35:20.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj13 td/osd-recovery-space/datafile 2024-04-25T08:35:20.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj14 td/osd-recovery-space/datafile 2024-04-25T08:35:20.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj15 td/osd-recovery-space/datafile 2024-04-25T08:35:20.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj16 td/osd-recovery-space/datafile 2024-04-25T08:35:20.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj17 td/osd-recovery-space/datafile 2024-04-25T08:35:20.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj18 td/osd-recovery-space/datafile 2024-04-25T08:35:20.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:20.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj19 td/osd-recovery-space/datafile 2024-04-25T08:35:21.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:21.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj20 td/osd-recovery-space/datafile 2024-04-25T08:35:21.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:21.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj21 td/osd-recovery-space/datafile 2024-04-25T08:35:21.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:21.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj22 td/osd-recovery-space/datafile 2024-04-25T08:35:21.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:21.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj23 td/osd-recovery-space/datafile 2024-04-25T08:35:21.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:21.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj24 td/osd-recovery-space/datafile 2024-04-25T08:35:21.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:21.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj25 td/osd-recovery-space/datafile 2024-04-25T08:35:21.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:21.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj26 td/osd-recovery-space/datafile 2024-04-25T08:35:21.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:21.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj27 td/osd-recovery-space/datafile 2024-04-25T08:35:21.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:21.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj28 td/osd-recovery-space/datafile 2024-04-25T08:35:22.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:22.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj29 td/osd-recovery-space/datafile 2024-04-25T08:35:22.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:22.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj30 td/osd-recovery-space/datafile 2024-04-25T08:35:22.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:22.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj31 td/osd-recovery-space/datafile 2024-04-25T08:35:22.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:22.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj32 td/osd-recovery-space/datafile 2024-04-25T08:35:22.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:22.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj33 td/osd-recovery-space/datafile 2024-04-25T08:35:22.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:22.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj34 td/osd-recovery-space/datafile 2024-04-25T08:35:22.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:22.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj35 td/osd-recovery-space/datafile 2024-04-25T08:35:22.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:22.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj36 td/osd-recovery-space/datafile 2024-04-25T08:35:22.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:22.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj37 td/osd-recovery-space/datafile 2024-04-25T08:35:23.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj38 td/osd-recovery-space/datafile 2024-04-25T08:35:23.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj39 td/osd-recovery-space/datafile 2024-04-25T08:35:23.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj40 td/osd-recovery-space/datafile 2024-04-25T08:35:23.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj41 td/osd-recovery-space/datafile 2024-04-25T08:35:23.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj42 td/osd-recovery-space/datafile 2024-04-25T08:35:23.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj43 td/osd-recovery-space/datafile 2024-04-25T08:35:23.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj44 td/osd-recovery-space/datafile 2024-04-25T08:35:23.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj45 td/osd-recovery-space/datafile 2024-04-25T08:35:23.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj46 td/osd-recovery-space/datafile 2024-04-25T08:35:23.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:23.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj47 td/osd-recovery-space/datafile 2024-04-25T08:35:24.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:24.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj48 td/osd-recovery-space/datafile 2024-04-25T08:35:24.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:24.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj49 td/osd-recovery-space/datafile 2024-04-25T08:35:24.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:24.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj50 td/osd-recovery-space/datafile 2024-04-25T08:35:24.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:24.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj51 td/osd-recovery-space/datafile 2024-04-25T08:35:24.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:24.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj52 td/osd-recovery-space/datafile 2024-04-25T08:35:24.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:24.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj53 td/osd-recovery-space/datafile 2024-04-25T08:35:24.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:24.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj54 td/osd-recovery-space/datafile 2024-04-25T08:35:24.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:24.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj55 td/osd-recovery-space/datafile 2024-04-25T08:35:25.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:25.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj56 td/osd-recovery-space/datafile 2024-04-25T08:35:25.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:25.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj57 td/osd-recovery-space/datafile 2024-04-25T08:35:25.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:25.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj58 td/osd-recovery-space/datafile 2024-04-25T08:35:25.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:25.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj59 td/osd-recovery-space/datafile 2024-04-25T08:35:25.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:25.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj60 td/osd-recovery-space/datafile 2024-04-25T08:35:25.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:25.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj61 td/osd-recovery-space/datafile 2024-04-25T08:35:25.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:25.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj62 td/osd-recovery-space/datafile 2024-04-25T08:35:25.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:25.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj63 td/osd-recovery-space/datafile 2024-04-25T08:35:26.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:26.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj64 td/osd-recovery-space/datafile 2024-04-25T08:35:26.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:26.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj65 td/osd-recovery-space/datafile 2024-04-25T08:35:26.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:26.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj66 td/osd-recovery-space/datafile 2024-04-25T08:35:26.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:26.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj67 td/osd-recovery-space/datafile 2024-04-25T08:35:26.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:26.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj68 td/osd-recovery-space/datafile 2024-04-25T08:35:26.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:26.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj69 td/osd-recovery-space/datafile 2024-04-25T08:35:26.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:26.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj70 td/osd-recovery-space/datafile 2024-04-25T08:35:26.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:26.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj71 td/osd-recovery-space/datafile 2024-04-25T08:35:26.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:26.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj72 td/osd-recovery-space/datafile 2024-04-25T08:35:27.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:27.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj73 td/osd-recovery-space/datafile 2024-04-25T08:35:27.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:27.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj74 td/osd-recovery-space/datafile 2024-04-25T08:35:27.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:27.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj75 td/osd-recovery-space/datafile 2024-04-25T08:35:27.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:27.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj76 td/osd-recovery-space/datafile 2024-04-25T08:35:27.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:27.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj77 td/osd-recovery-space/datafile 2024-04-25T08:35:27.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:27.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj78 td/osd-recovery-space/datafile 2024-04-25T08:35:27.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:27.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj79 td/osd-recovery-space/datafile 2024-04-25T08:35:27.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:27.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj80 td/osd-recovery-space/datafile 2024-04-25T08:35:27.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:27.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj81 td/osd-recovery-space/datafile 2024-04-25T08:35:28.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:28.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj82 td/osd-recovery-space/datafile 2024-04-25T08:35:28.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:28.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj83 td/osd-recovery-space/datafile 2024-04-25T08:35:28.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:28.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj84 td/osd-recovery-space/datafile 2024-04-25T08:35:28.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:28.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj85 td/osd-recovery-space/datafile 2024-04-25T08:35:28.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:28.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj86 td/osd-recovery-space/datafile 2024-04-25T08:35:28.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:28.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj87 td/osd-recovery-space/datafile 2024-04-25T08:35:28.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:28.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj88 td/osd-recovery-space/datafile 2024-04-25T08:35:28.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:28.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj89 td/osd-recovery-space/datafile 2024-04-25T08:35:28.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:28.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj90 td/osd-recovery-space/datafile 2024-04-25T08:35:29.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj91 td/osd-recovery-space/datafile 2024-04-25T08:35:29.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj92 td/osd-recovery-space/datafile 2024-04-25T08:35:29.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj93 td/osd-recovery-space/datafile 2024-04-25T08:35:29.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj94 td/osd-recovery-space/datafile 2024-04-25T08:35:29.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj95 td/osd-recovery-space/datafile 2024-04-25T08:35:29.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj96 td/osd-recovery-space/datafile 2024-04-25T08:35:29.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj97 td/osd-recovery-space/datafile 2024-04-25T08:35:29.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj98 td/osd-recovery-space/datafile 2024-04-25T08:35:29.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj99 td/osd-recovery-space/datafile 2024-04-25T08:35:29.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:29.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj100 td/osd-recovery-space/datafile 2024-04-25T08:35:30.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:30.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj101 td/osd-recovery-space/datafile 2024-04-25T08:35:30.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:30.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj102 td/osd-recovery-space/datafile 2024-04-25T08:35:30.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:30.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj103 td/osd-recovery-space/datafile 2024-04-25T08:35:30.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:30.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj104 td/osd-recovery-space/datafile 2024-04-25T08:35:30.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:30.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj105 td/osd-recovery-space/datafile 2024-04-25T08:35:30.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:30.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj106 td/osd-recovery-space/datafile 2024-04-25T08:35:30.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:30.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj107 td/osd-recovery-space/datafile 2024-04-25T08:35:30.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:30.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj108 td/osd-recovery-space/datafile 2024-04-25T08:35:30.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:30.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj109 td/osd-recovery-space/datafile 2024-04-25T08:35:31.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:31.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj110 td/osd-recovery-space/datafile 2024-04-25T08:35:31.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:31.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj111 td/osd-recovery-space/datafile 2024-04-25T08:35:31.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:31.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj112 td/osd-recovery-space/datafile 2024-04-25T08:35:31.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:31.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj113 td/osd-recovery-space/datafile 2024-04-25T08:35:31.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:31.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj114 td/osd-recovery-space/datafile 2024-04-25T08:35:31.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:31.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj115 td/osd-recovery-space/datafile 2024-04-25T08:35:31.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:31.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj116 td/osd-recovery-space/datafile 2024-04-25T08:35:31.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:31.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj117 td/osd-recovery-space/datafile 2024-04-25T08:35:31.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:31.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj118 td/osd-recovery-space/datafile 2024-04-25T08:35:32.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:32.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj119 td/osd-recovery-space/datafile 2024-04-25T08:35:32.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:32.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj120 td/osd-recovery-space/datafile 2024-04-25T08:35:32.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:32.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj121 td/osd-recovery-space/datafile 2024-04-25T08:35:32.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:32.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj122 td/osd-recovery-space/datafile 2024-04-25T08:35:32.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:32.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj123 td/osd-recovery-space/datafile 2024-04-25T08:35:32.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:32.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj124 td/osd-recovery-space/datafile 2024-04-25T08:35:32.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:32.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj125 td/osd-recovery-space/datafile 2024-04-25T08:35:32.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:32.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj126 td/osd-recovery-space/datafile 2024-04-25T08:35:32.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:32.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj127 td/osd-recovery-space/datafile 2024-04-25T08:35:33.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:33.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj128 td/osd-recovery-space/datafile 2024-04-25T08:35:33.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:33.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj129 td/osd-recovery-space/datafile 2024-04-25T08:35:33.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:33.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj130 td/osd-recovery-space/datafile 2024-04-25T08:35:33.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:33.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj131 td/osd-recovery-space/datafile 2024-04-25T08:35:33.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:33.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj132 td/osd-recovery-space/datafile 2024-04-25T08:35:33.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:33.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj133 td/osd-recovery-space/datafile 2024-04-25T08:35:33.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:33.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj134 td/osd-recovery-space/datafile 2024-04-25T08:35:33.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:33.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj135 td/osd-recovery-space/datafile 2024-04-25T08:35:33.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:33.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj136 td/osd-recovery-space/datafile 2024-04-25T08:35:34.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:34.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj137 td/osd-recovery-space/datafile 2024-04-25T08:35:34.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:34.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj138 td/osd-recovery-space/datafile 2024-04-25T08:35:34.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:34.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj139 td/osd-recovery-space/datafile 2024-04-25T08:35:34.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:34.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj140 td/osd-recovery-space/datafile 2024-04-25T08:35:34.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:34.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj141 td/osd-recovery-space/datafile 2024-04-25T08:35:34.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:34.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj142 td/osd-recovery-space/datafile 2024-04-25T08:35:34.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:34.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj143 td/osd-recovery-space/datafile 2024-04-25T08:35:34.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:34.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj144 td/osd-recovery-space/datafile 2024-04-25T08:35:34.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:34.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj145 td/osd-recovery-space/datafile 2024-04-25T08:35:35.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:35.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj146 td/osd-recovery-space/datafile 2024-04-25T08:35:35.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:35.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj147 td/osd-recovery-space/datafile 2024-04-25T08:35:35.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:35.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj148 td/osd-recovery-space/datafile 2024-04-25T08:35:35.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:35.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj149 td/osd-recovery-space/datafile 2024-04-25T08:35:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj150 td/osd-recovery-space/datafile 2024-04-25T08:35:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj151 td/osd-recovery-space/datafile 2024-04-25T08:35:35.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:35.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj152 td/osd-recovery-space/datafile 2024-04-25T08:35:35.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:35.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj153 td/osd-recovery-space/datafile 2024-04-25T08:35:35.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:35.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj154 td/osd-recovery-space/datafile 2024-04-25T08:35:36.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:36.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj155 td/osd-recovery-space/datafile 2024-04-25T08:35:36.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:36.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj156 td/osd-recovery-space/datafile 2024-04-25T08:35:36.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:36.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj157 td/osd-recovery-space/datafile 2024-04-25T08:35:36.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:36.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj158 td/osd-recovery-space/datafile 2024-04-25T08:35:36.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:36.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj159 td/osd-recovery-space/datafile 2024-04-25T08:35:36.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:36.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj160 td/osd-recovery-space/datafile 2024-04-25T08:35:36.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:36.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj161 td/osd-recovery-space/datafile 2024-04-25T08:35:36.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:36.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj162 td/osd-recovery-space/datafile 2024-04-25T08:35:36.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:36.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj163 td/osd-recovery-space/datafile 2024-04-25T08:35:37.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:37.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj164 td/osd-recovery-space/datafile 2024-04-25T08:35:37.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:37.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj165 td/osd-recovery-space/datafile 2024-04-25T08:35:37.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:37.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj166 td/osd-recovery-space/datafile 2024-04-25T08:35:37.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:37.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj167 td/osd-recovery-space/datafile 2024-04-25T08:35:37.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:37.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj168 td/osd-recovery-space/datafile 2024-04-25T08:35:37.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:37.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj169 td/osd-recovery-space/datafile 2024-04-25T08:35:37.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:37.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj170 td/osd-recovery-space/datafile 2024-04-25T08:35:37.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:37.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj171 td/osd-recovery-space/datafile 2024-04-25T08:35:37.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:37.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj172 td/osd-recovery-space/datafile 2024-04-25T08:35:38.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:38.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj173 td/osd-recovery-space/datafile 2024-04-25T08:35:38.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:38.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj174 td/osd-recovery-space/datafile 2024-04-25T08:35:38.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:38.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj175 td/osd-recovery-space/datafile 2024-04-25T08:35:38.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:38.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj176 td/osd-recovery-space/datafile 2024-04-25T08:35:38.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:38.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj177 td/osd-recovery-space/datafile 2024-04-25T08:35:38.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:38.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj178 td/osd-recovery-space/datafile 2024-04-25T08:35:38.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:38.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj179 td/osd-recovery-space/datafile 2024-04-25T08:35:38.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:38.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj180 td/osd-recovery-space/datafile 2024-04-25T08:35:38.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:38.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj181 td/osd-recovery-space/datafile 2024-04-25T08:35:39.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj182 td/osd-recovery-space/datafile 2024-04-25T08:35:39.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj183 td/osd-recovery-space/datafile 2024-04-25T08:35:39.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj184 td/osd-recovery-space/datafile 2024-04-25T08:35:39.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj185 td/osd-recovery-space/datafile 2024-04-25T08:35:39.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj186 td/osd-recovery-space/datafile 2024-04-25T08:35:39.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj187 td/osd-recovery-space/datafile 2024-04-25T08:35:39.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj188 td/osd-recovery-space/datafile 2024-04-25T08:35:39.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj189 td/osd-recovery-space/datafile 2024-04-25T08:35:39.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj190 td/osd-recovery-space/datafile 2024-04-25T08:35:39.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:39.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj191 td/osd-recovery-space/datafile 2024-04-25T08:35:40.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:40.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj192 td/osd-recovery-space/datafile 2024-04-25T08:35:40.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:40.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj193 td/osd-recovery-space/datafile 2024-04-25T08:35:40.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:40.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj194 td/osd-recovery-space/datafile 2024-04-25T08:35:40.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:40.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj195 td/osd-recovery-space/datafile 2024-04-25T08:35:40.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:40.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj196 td/osd-recovery-space/datafile 2024-04-25T08:35:40.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:40.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj197 td/osd-recovery-space/datafile 2024-04-25T08:35:40.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:40.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj198 td/osd-recovery-space/datafile 2024-04-25T08:35:40.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:40.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj199 td/osd-recovery-space/datafile 2024-04-25T08:35:41.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj200 td/osd-recovery-space/datafile 2024-04-25T08:35:41.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj201 td/osd-recovery-space/datafile 2024-04-25T08:35:41.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj202 td/osd-recovery-space/datafile 2024-04-25T08:35:41.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj203 td/osd-recovery-space/datafile 2024-04-25T08:35:41.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj204 td/osd-recovery-space/datafile 2024-04-25T08:35:41.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj205 td/osd-recovery-space/datafile 2024-04-25T08:35:41.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj206 td/osd-recovery-space/datafile 2024-04-25T08:35:41.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj207 td/osd-recovery-space/datafile 2024-04-25T08:35:41.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj208 td/osd-recovery-space/datafile 2024-04-25T08:35:41.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:41.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj209 td/osd-recovery-space/datafile 2024-04-25T08:35:42.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:42.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj210 td/osd-recovery-space/datafile 2024-04-25T08:35:42.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:42.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj211 td/osd-recovery-space/datafile 2024-04-25T08:35:42.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:42.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj212 td/osd-recovery-space/datafile 2024-04-25T08:35:42.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:42.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj213 td/osd-recovery-space/datafile 2024-04-25T08:35:42.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:42.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj214 td/osd-recovery-space/datafile 2024-04-25T08:35:42.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:42.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj215 td/osd-recovery-space/datafile 2024-04-25T08:35:42.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:42.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj216 td/osd-recovery-space/datafile 2024-04-25T08:35:42.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:42.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj217 td/osd-recovery-space/datafile 2024-04-25T08:35:43.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:43.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj218 td/osd-recovery-space/datafile 2024-04-25T08:35:43.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:43.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj219 td/osd-recovery-space/datafile 2024-04-25T08:35:43.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:43.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj220 td/osd-recovery-space/datafile 2024-04-25T08:35:43.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:43.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj221 td/osd-recovery-space/datafile 2024-04-25T08:35:43.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:43.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj222 td/osd-recovery-space/datafile 2024-04-25T08:35:43.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:43.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj223 td/osd-recovery-space/datafile 2024-04-25T08:35:43.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:43.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj224 td/osd-recovery-space/datafile 2024-04-25T08:35:43.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:43.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj225 td/osd-recovery-space/datafile 2024-04-25T08:35:43.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:43.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj226 td/osd-recovery-space/datafile 2024-04-25T08:35:44.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj227 td/osd-recovery-space/datafile 2024-04-25T08:35:44.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj228 td/osd-recovery-space/datafile 2024-04-25T08:35:44.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj229 td/osd-recovery-space/datafile 2024-04-25T08:35:44.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj230 td/osd-recovery-space/datafile 2024-04-25T08:35:44.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj231 td/osd-recovery-space/datafile 2024-04-25T08:35:44.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj232 td/osd-recovery-space/datafile 2024-04-25T08:35:44.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj233 td/osd-recovery-space/datafile 2024-04-25T08:35:44.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj234 td/osd-recovery-space/datafile 2024-04-25T08:35:44.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj235 td/osd-recovery-space/datafile 2024-04-25T08:35:44.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:44.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj236 td/osd-recovery-space/datafile 2024-04-25T08:35:45.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:45.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj237 td/osd-recovery-space/datafile 2024-04-25T08:35:45.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:45.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj238 td/osd-recovery-space/datafile 2024-04-25T08:35:45.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:45.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj239 td/osd-recovery-space/datafile 2024-04-25T08:35:45.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:45.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj240 td/osd-recovery-space/datafile 2024-04-25T08:35:45.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:45.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj241 td/osd-recovery-space/datafile 2024-04-25T08:35:45.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:45.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj242 td/osd-recovery-space/datafile 2024-04-25T08:35:45.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:45.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj243 td/osd-recovery-space/datafile 2024-04-25T08:35:45.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:45.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj244 td/osd-recovery-space/datafile 2024-04-25T08:35:46.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:46.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj245 td/osd-recovery-space/datafile 2024-04-25T08:35:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj246 td/osd-recovery-space/datafile 2024-04-25T08:35:46.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:46.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj247 td/osd-recovery-space/datafile 2024-04-25T08:35:46.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:46.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj248 td/osd-recovery-space/datafile 2024-04-25T08:35:46.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:46.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj249 td/osd-recovery-space/datafile 2024-04-25T08:35:46.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:46.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj250 td/osd-recovery-space/datafile 2024-04-25T08:35:46.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:46.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj251 td/osd-recovery-space/datafile 2024-04-25T08:35:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj252 td/osd-recovery-space/datafile 2024-04-25T08:35:46.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:46.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj253 td/osd-recovery-space/datafile 2024-04-25T08:35:47.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj254 td/osd-recovery-space/datafile 2024-04-25T08:35:47.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj255 td/osd-recovery-space/datafile 2024-04-25T08:35:47.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj256 td/osd-recovery-space/datafile 2024-04-25T08:35:47.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj257 td/osd-recovery-space/datafile 2024-04-25T08:35:47.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj258 td/osd-recovery-space/datafile 2024-04-25T08:35:47.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj259 td/osd-recovery-space/datafile 2024-04-25T08:35:47.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj260 td/osd-recovery-space/datafile 2024-04-25T08:35:47.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj261 td/osd-recovery-space/datafile 2024-04-25T08:35:47.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj262 td/osd-recovery-space/datafile 2024-04-25T08:35:47.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:47.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj263 td/osd-recovery-space/datafile 2024-04-25T08:35:48.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:48.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj264 td/osd-recovery-space/datafile 2024-04-25T08:35:48.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:48.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj265 td/osd-recovery-space/datafile 2024-04-25T08:35:48.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:48.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj266 td/osd-recovery-space/datafile 2024-04-25T08:35:48.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:48.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj267 td/osd-recovery-space/datafile 2024-04-25T08:35:48.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:48.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj268 td/osd-recovery-space/datafile 2024-04-25T08:35:48.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:48.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj269 td/osd-recovery-space/datafile 2024-04-25T08:35:48.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:48.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj270 td/osd-recovery-space/datafile 2024-04-25T08:35:48.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:48.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj271 td/osd-recovery-space/datafile 2024-04-25T08:35:48.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:48.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj272 td/osd-recovery-space/datafile 2024-04-25T08:35:49.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:49.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj273 td/osd-recovery-space/datafile 2024-04-25T08:35:49.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:49.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj274 td/osd-recovery-space/datafile 2024-04-25T08:35:49.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:49.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj275 td/osd-recovery-space/datafile 2024-04-25T08:35:49.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:49.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj276 td/osd-recovery-space/datafile 2024-04-25T08:35:49.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:49.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj277 td/osd-recovery-space/datafile 2024-04-25T08:35:49.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:49.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj278 td/osd-recovery-space/datafile 2024-04-25T08:35:49.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:49.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj279 td/osd-recovery-space/datafile 2024-04-25T08:35:49.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:49.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj280 td/osd-recovery-space/datafile 2024-04-25T08:35:49.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:49.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj281 td/osd-recovery-space/datafile 2024-04-25T08:35:50.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:50.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj282 td/osd-recovery-space/datafile 2024-04-25T08:35:50.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:50.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj283 td/osd-recovery-space/datafile 2024-04-25T08:35:50.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:50.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj284 td/osd-recovery-space/datafile 2024-04-25T08:35:50.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:50.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj285 td/osd-recovery-space/datafile 2024-04-25T08:35:50.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:50.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj286 td/osd-recovery-space/datafile 2024-04-25T08:35:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj287 td/osd-recovery-space/datafile 2024-04-25T08:35:50.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:50.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj288 td/osd-recovery-space/datafile 2024-04-25T08:35:50.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:50.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj289 td/osd-recovery-space/datafile 2024-04-25T08:35:50.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:50.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj290 td/osd-recovery-space/datafile 2024-04-25T08:35:51.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:51.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj291 td/osd-recovery-space/datafile 2024-04-25T08:35:51.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:51.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj292 td/osd-recovery-space/datafile 2024-04-25T08:35:51.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:51.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj293 td/osd-recovery-space/datafile 2024-04-25T08:35:51.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:51.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj294 td/osd-recovery-space/datafile 2024-04-25T08:35:51.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:51.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj295 td/osd-recovery-space/datafile 2024-04-25T08:35:51.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:51.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj296 td/osd-recovery-space/datafile 2024-04-25T08:35:51.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:51.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj297 td/osd-recovery-space/datafile 2024-04-25T08:35:51.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:51.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj298 td/osd-recovery-space/datafile 2024-04-25T08:35:51.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:51.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj299 td/osd-recovery-space/datafile 2024-04-25T08:35:52.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:52.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj300 td/osd-recovery-space/datafile 2024-04-25T08:35:52.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:52.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj301 td/osd-recovery-space/datafile 2024-04-25T08:35:52.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:52.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj302 td/osd-recovery-space/datafile 2024-04-25T08:35:52.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:52.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj303 td/osd-recovery-space/datafile 2024-04-25T08:35:52.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:52.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj304 td/osd-recovery-space/datafile 2024-04-25T08:35:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj305 td/osd-recovery-space/datafile 2024-04-25T08:35:52.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:52.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj306 td/osd-recovery-space/datafile 2024-04-25T08:35:52.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:52.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj307 td/osd-recovery-space/datafile 2024-04-25T08:35:52.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:52.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj308 td/osd-recovery-space/datafile 2024-04-25T08:35:53.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:53.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj309 td/osd-recovery-space/datafile 2024-04-25T08:35:53.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:53.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj310 td/osd-recovery-space/datafile 2024-04-25T08:35:53.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:53.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj311 td/osd-recovery-space/datafile 2024-04-25T08:35:53.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:53.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj312 td/osd-recovery-space/datafile 2024-04-25T08:35:53.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:53.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj313 td/osd-recovery-space/datafile 2024-04-25T08:35:53.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:53.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj314 td/osd-recovery-space/datafile 2024-04-25T08:35:53.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:53.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj315 td/osd-recovery-space/datafile 2024-04-25T08:35:53.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:53.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj316 td/osd-recovery-space/datafile 2024-04-25T08:35:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj317 td/osd-recovery-space/datafile 2024-04-25T08:35:54.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:54.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj318 td/osd-recovery-space/datafile 2024-04-25T08:35:54.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:54.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj319 td/osd-recovery-space/datafile 2024-04-25T08:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj320 td/osd-recovery-space/datafile 2024-04-25T08:35:54.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:54.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj321 td/osd-recovery-space/datafile 2024-04-25T08:35:54.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:54.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj322 td/osd-recovery-space/datafile 2024-04-25T08:35:54.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:54.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj323 td/osd-recovery-space/datafile 2024-04-25T08:35:54.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:54.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj324 td/osd-recovery-space/datafile 2024-04-25T08:35:54.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:54.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj325 td/osd-recovery-space/datafile 2024-04-25T08:35:54.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:54.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj326 td/osd-recovery-space/datafile 2024-04-25T08:35:55.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj327 td/osd-recovery-space/datafile 2024-04-25T08:35:55.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj328 td/osd-recovery-space/datafile 2024-04-25T08:35:55.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj329 td/osd-recovery-space/datafile 2024-04-25T08:35:55.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj330 td/osd-recovery-space/datafile 2024-04-25T08:35:55.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj331 td/osd-recovery-space/datafile 2024-04-25T08:35:55.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj332 td/osd-recovery-space/datafile 2024-04-25T08:35:55.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj333 td/osd-recovery-space/datafile 2024-04-25T08:35:55.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj334 td/osd-recovery-space/datafile 2024-04-25T08:35:55.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj335 td/osd-recovery-space/datafile 2024-04-25T08:35:55.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj336 td/osd-recovery-space/datafile 2024-04-25T08:35:55.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj337 td/osd-recovery-space/datafile 2024-04-25T08:35:55.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:55.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj338 td/osd-recovery-space/datafile 2024-04-25T08:35:56.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:56.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj339 td/osd-recovery-space/datafile 2024-04-25T08:35:56.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:56.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj340 td/osd-recovery-space/datafile 2024-04-25T08:35:56.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:56.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj341 td/osd-recovery-space/datafile 2024-04-25T08:35:56.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:56.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj342 td/osd-recovery-space/datafile 2024-04-25T08:35:56.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:56.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj343 td/osd-recovery-space/datafile 2024-04-25T08:35:56.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:56.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj344 td/osd-recovery-space/datafile 2024-04-25T08:35:56.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:56.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj345 td/osd-recovery-space/datafile 2024-04-25T08:35:56.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:56.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj346 td/osd-recovery-space/datafile 2024-04-25T08:35:56.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:56.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj347 td/osd-recovery-space/datafile 2024-04-25T08:35:57.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj348 td/osd-recovery-space/datafile 2024-04-25T08:35:57.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj349 td/osd-recovery-space/datafile 2024-04-25T08:35:57.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj350 td/osd-recovery-space/datafile 2024-04-25T08:35:57.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj351 td/osd-recovery-space/datafile 2024-04-25T08:35:57.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj352 td/osd-recovery-space/datafile 2024-04-25T08:35:57.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj353 td/osd-recovery-space/datafile 2024-04-25T08:35:57.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj354 td/osd-recovery-space/datafile 2024-04-25T08:35:57.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj355 td/osd-recovery-space/datafile 2024-04-25T08:35:57.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj356 td/osd-recovery-space/datafile 2024-04-25T08:35:57.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj357 td/osd-recovery-space/datafile 2024-04-25T08:35:57.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj358 td/osd-recovery-space/datafile 2024-04-25T08:35:57.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:57.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj359 td/osd-recovery-space/datafile 2024-04-25T08:35:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj360 td/osd-recovery-space/datafile 2024-04-25T08:35:58.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj361 td/osd-recovery-space/datafile 2024-04-25T08:35:58.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj362 td/osd-recovery-space/datafile 2024-04-25T08:35:58.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj363 td/osd-recovery-space/datafile 2024-04-25T08:35:58.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj364 td/osd-recovery-space/datafile 2024-04-25T08:35:58.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj365 td/osd-recovery-space/datafile 2024-04-25T08:35:58.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj366 td/osd-recovery-space/datafile 2024-04-25T08:35:58.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj367 td/osd-recovery-space/datafile 2024-04-25T08:35:58.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj368 td/osd-recovery-space/datafile 2024-04-25T08:35:58.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj369 td/osd-recovery-space/datafile 2024-04-25T08:35:58.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj370 td/osd-recovery-space/datafile 2024-04-25T08:35:58.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:58.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj371 td/osd-recovery-space/datafile 2024-04-25T08:35:59.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj372 td/osd-recovery-space/datafile 2024-04-25T08:35:59.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj373 td/osd-recovery-space/datafile 2024-04-25T08:35:59.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj374 td/osd-recovery-space/datafile 2024-04-25T08:35:59.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj375 td/osd-recovery-space/datafile 2024-04-25T08:35:59.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj376 td/osd-recovery-space/datafile 2024-04-25T08:35:59.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj377 td/osd-recovery-space/datafile 2024-04-25T08:35:59.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj378 td/osd-recovery-space/datafile 2024-04-25T08:35:59.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj379 td/osd-recovery-space/datafile 2024-04-25T08:35:59.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj380 td/osd-recovery-space/datafile 2024-04-25T08:35:59.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj381 td/osd-recovery-space/datafile 2024-04-25T08:35:59.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj382 td/osd-recovery-space/datafile 2024-04-25T08:35:59.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj383 td/osd-recovery-space/datafile 2024-04-25T08:35:59.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:35:59.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj384 td/osd-recovery-space/datafile 2024-04-25T08:36:00.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj385 td/osd-recovery-space/datafile 2024-04-25T08:36:00.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj386 td/osd-recovery-space/datafile 2024-04-25T08:36:00.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj387 td/osd-recovery-space/datafile 2024-04-25T08:36:00.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj388 td/osd-recovery-space/datafile 2024-04-25T08:36:00.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj389 td/osd-recovery-space/datafile 2024-04-25T08:36:00.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj390 td/osd-recovery-space/datafile 2024-04-25T08:36:00.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj391 td/osd-recovery-space/datafile 2024-04-25T08:36:00.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj392 td/osd-recovery-space/datafile 2024-04-25T08:36:00.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj393 td/osd-recovery-space/datafile 2024-04-25T08:36:00.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj394 td/osd-recovery-space/datafile 2024-04-25T08:36:00.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:00.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj395 td/osd-recovery-space/datafile 2024-04-25T08:36:01.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj396 td/osd-recovery-space/datafile 2024-04-25T08:36:01.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj397 td/osd-recovery-space/datafile 2024-04-25T08:36:01.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj398 td/osd-recovery-space/datafile 2024-04-25T08:36:01.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj399 td/osd-recovery-space/datafile 2024-04-25T08:36:01.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj400 td/osd-recovery-space/datafile 2024-04-25T08:36:01.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj401 td/osd-recovery-space/datafile 2024-04-25T08:36:01.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj402 td/osd-recovery-space/datafile 2024-04-25T08:36:01.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj403 td/osd-recovery-space/datafile 2024-04-25T08:36:01.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj404 td/osd-recovery-space/datafile 2024-04-25T08:36:01.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj405 td/osd-recovery-space/datafile 2024-04-25T08:36:01.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj406 td/osd-recovery-space/datafile 2024-04-25T08:36:01.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:01.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj407 td/osd-recovery-space/datafile 2024-04-25T08:36:02.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj408 td/osd-recovery-space/datafile 2024-04-25T08:36:02.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj409 td/osd-recovery-space/datafile 2024-04-25T08:36:02.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj410 td/osd-recovery-space/datafile 2024-04-25T08:36:02.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj411 td/osd-recovery-space/datafile 2024-04-25T08:36:02.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj412 td/osd-recovery-space/datafile 2024-04-25T08:36:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj413 td/osd-recovery-space/datafile 2024-04-25T08:36:02.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj414 td/osd-recovery-space/datafile 2024-04-25T08:36:02.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj415 td/osd-recovery-space/datafile 2024-04-25T08:36:02.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj416 td/osd-recovery-space/datafile 2024-04-25T08:36:02.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj417 td/osd-recovery-space/datafile 2024-04-25T08:36:02.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj418 td/osd-recovery-space/datafile 2024-04-25T08:36:02.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj419 td/osd-recovery-space/datafile 2024-04-25T08:36:02.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:02.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj420 td/osd-recovery-space/datafile 2024-04-25T08:36:03.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj421 td/osd-recovery-space/datafile 2024-04-25T08:36:03.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj422 td/osd-recovery-space/datafile 2024-04-25T08:36:03.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj423 td/osd-recovery-space/datafile 2024-04-25T08:36:03.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj424 td/osd-recovery-space/datafile 2024-04-25T08:36:03.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj425 td/osd-recovery-space/datafile 2024-04-25T08:36:03.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj426 td/osd-recovery-space/datafile 2024-04-25T08:36:03.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj427 td/osd-recovery-space/datafile 2024-04-25T08:36:03.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj428 td/osd-recovery-space/datafile 2024-04-25T08:36:03.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj429 td/osd-recovery-space/datafile 2024-04-25T08:36:03.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj430 td/osd-recovery-space/datafile 2024-04-25T08:36:03.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj431 td/osd-recovery-space/datafile 2024-04-25T08:36:03.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj432 td/osd-recovery-space/datafile 2024-04-25T08:36:03.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:03.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj433 td/osd-recovery-space/datafile 2024-04-25T08:36:04.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj434 td/osd-recovery-space/datafile 2024-04-25T08:36:04.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj435 td/osd-recovery-space/datafile 2024-04-25T08:36:04.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj436 td/osd-recovery-space/datafile 2024-04-25T08:36:04.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj437 td/osd-recovery-space/datafile 2024-04-25T08:36:04.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj438 td/osd-recovery-space/datafile 2024-04-25T08:36:04.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj439 td/osd-recovery-space/datafile 2024-04-25T08:36:04.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj440 td/osd-recovery-space/datafile 2024-04-25T08:36:04.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj441 td/osd-recovery-space/datafile 2024-04-25T08:36:04.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj442 td/osd-recovery-space/datafile 2024-04-25T08:36:04.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj443 td/osd-recovery-space/datafile 2024-04-25T08:36:04.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:04.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj444 td/osd-recovery-space/datafile 2024-04-25T08:36:05.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj445 td/osd-recovery-space/datafile 2024-04-25T08:36:05.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj446 td/osd-recovery-space/datafile 2024-04-25T08:36:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj447 td/osd-recovery-space/datafile 2024-04-25T08:36:05.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj448 td/osd-recovery-space/datafile 2024-04-25T08:36:05.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj449 td/osd-recovery-space/datafile 2024-04-25T08:36:05.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj450 td/osd-recovery-space/datafile 2024-04-25T08:36:05.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj451 td/osd-recovery-space/datafile 2024-04-25T08:36:05.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj452 td/osd-recovery-space/datafile 2024-04-25T08:36:05.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj453 td/osd-recovery-space/datafile 2024-04-25T08:36:05.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj454 td/osd-recovery-space/datafile 2024-04-25T08:36:05.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj455 td/osd-recovery-space/datafile 2024-04-25T08:36:05.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj456 td/osd-recovery-space/datafile 2024-04-25T08:36:05.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:05.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj457 td/osd-recovery-space/datafile 2024-04-25T08:36:06.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj458 td/osd-recovery-space/datafile 2024-04-25T08:36:06.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj459 td/osd-recovery-space/datafile 2024-04-25T08:36:06.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj460 td/osd-recovery-space/datafile 2024-04-25T08:36:06.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj461 td/osd-recovery-space/datafile 2024-04-25T08:36:06.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj462 td/osd-recovery-space/datafile 2024-04-25T08:36:06.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj463 td/osd-recovery-space/datafile 2024-04-25T08:36:06.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj464 td/osd-recovery-space/datafile 2024-04-25T08:36:06.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj465 td/osd-recovery-space/datafile 2024-04-25T08:36:06.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj466 td/osd-recovery-space/datafile 2024-04-25T08:36:06.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj467 td/osd-recovery-space/datafile 2024-04-25T08:36:06.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:06.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj468 td/osd-recovery-space/datafile 2024-04-25T08:36:07.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj469 td/osd-recovery-space/datafile 2024-04-25T08:36:07.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj470 td/osd-recovery-space/datafile 2024-04-25T08:36:07.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj471 td/osd-recovery-space/datafile 2024-04-25T08:36:07.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj472 td/osd-recovery-space/datafile 2024-04-25T08:36:07.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj473 td/osd-recovery-space/datafile 2024-04-25T08:36:07.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj474 td/osd-recovery-space/datafile 2024-04-25T08:36:07.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj475 td/osd-recovery-space/datafile 2024-04-25T08:36:07.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj476 td/osd-recovery-space/datafile 2024-04-25T08:36:07.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj477 td/osd-recovery-space/datafile 2024-04-25T08:36:07.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj478 td/osd-recovery-space/datafile 2024-04-25T08:36:07.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj479 td/osd-recovery-space/datafile 2024-04-25T08:36:07.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj480 td/osd-recovery-space/datafile 2024-04-25T08:36:07.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:07.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj481 td/osd-recovery-space/datafile 2024-04-25T08:36:08.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj482 td/osd-recovery-space/datafile 2024-04-25T08:36:08.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj483 td/osd-recovery-space/datafile 2024-04-25T08:36:08.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj484 td/osd-recovery-space/datafile 2024-04-25T08:36:08.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj485 td/osd-recovery-space/datafile 2024-04-25T08:36:08.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj486 td/osd-recovery-space/datafile 2024-04-25T08:36:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj487 td/osd-recovery-space/datafile 2024-04-25T08:36:08.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj488 td/osd-recovery-space/datafile 2024-04-25T08:36:08.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj489 td/osd-recovery-space/datafile 2024-04-25T08:36:08.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj490 td/osd-recovery-space/datafile 2024-04-25T08:36:08.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj491 td/osd-recovery-space/datafile 2024-04-25T08:36:08.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj492 td/osd-recovery-space/datafile 2024-04-25T08:36:08.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:08.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj493 td/osd-recovery-space/datafile 2024-04-25T08:36:09.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj494 td/osd-recovery-space/datafile 2024-04-25T08:36:09.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj495 td/osd-recovery-space/datafile 2024-04-25T08:36:09.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj496 td/osd-recovery-space/datafile 2024-04-25T08:36:09.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj497 td/osd-recovery-space/datafile 2024-04-25T08:36:09.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj498 td/osd-recovery-space/datafile 2024-04-25T08:36:09.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj499 td/osd-recovery-space/datafile 2024-04-25T08:36:09.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj500 td/osd-recovery-space/datafile 2024-04-25T08:36:09.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj501 td/osd-recovery-space/datafile 2024-04-25T08:36:09.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj502 td/osd-recovery-space/datafile 2024-04-25T08:36:09.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj503 td/osd-recovery-space/datafile 2024-04-25T08:36:09.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj504 td/osd-recovery-space/datafile 2024-04-25T08:36:09.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj505 td/osd-recovery-space/datafile 2024-04-25T08:36:09.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj506 td/osd-recovery-space/datafile 2024-04-25T08:36:09.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:09.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj507 td/osd-recovery-space/datafile 2024-04-25T08:36:10.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj508 td/osd-recovery-space/datafile 2024-04-25T08:36:10.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj509 td/osd-recovery-space/datafile 2024-04-25T08:36:10.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj510 td/osd-recovery-space/datafile 2024-04-25T08:36:10.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj511 td/osd-recovery-space/datafile 2024-04-25T08:36:10.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj512 td/osd-recovery-space/datafile 2024-04-25T08:36:10.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj513 td/osd-recovery-space/datafile 2024-04-25T08:36:10.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj514 td/osd-recovery-space/datafile 2024-04-25T08:36:10.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj515 td/osd-recovery-space/datafile 2024-04-25T08:36:10.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj516 td/osd-recovery-space/datafile 2024-04-25T08:36:10.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj517 td/osd-recovery-space/datafile 2024-04-25T08:36:10.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj518 td/osd-recovery-space/datafile 2024-04-25T08:36:10.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj519 td/osd-recovery-space/datafile 2024-04-25T08:36:10.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:10.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj520 td/osd-recovery-space/datafile 2024-04-25T08:36:11.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj521 td/osd-recovery-space/datafile 2024-04-25T08:36:11.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj522 td/osd-recovery-space/datafile 2024-04-25T08:36:11.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj523 td/osd-recovery-space/datafile 2024-04-25T08:36:11.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj524 td/osd-recovery-space/datafile 2024-04-25T08:36:11.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj525 td/osd-recovery-space/datafile 2024-04-25T08:36:11.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj526 td/osd-recovery-space/datafile 2024-04-25T08:36:11.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj527 td/osd-recovery-space/datafile 2024-04-25T08:36:11.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj528 td/osd-recovery-space/datafile 2024-04-25T08:36:11.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj529 td/osd-recovery-space/datafile 2024-04-25T08:36:11.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj530 td/osd-recovery-space/datafile 2024-04-25T08:36:11.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj531 td/osd-recovery-space/datafile 2024-04-25T08:36:11.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:11.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj532 td/osd-recovery-space/datafile 2024-04-25T08:36:12.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj533 td/osd-recovery-space/datafile 2024-04-25T08:36:12.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj534 td/osd-recovery-space/datafile 2024-04-25T08:36:12.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj535 td/osd-recovery-space/datafile 2024-04-25T08:36:12.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj536 td/osd-recovery-space/datafile 2024-04-25T08:36:12.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj537 td/osd-recovery-space/datafile 2024-04-25T08:36:12.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj538 td/osd-recovery-space/datafile 2024-04-25T08:36:12.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj539 td/osd-recovery-space/datafile 2024-04-25T08:36:12.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj540 td/osd-recovery-space/datafile 2024-04-25T08:36:12.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj541 td/osd-recovery-space/datafile 2024-04-25T08:36:12.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj542 td/osd-recovery-space/datafile 2024-04-25T08:36:12.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj543 td/osd-recovery-space/datafile 2024-04-25T08:36:12.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:12.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj544 td/osd-recovery-space/datafile 2024-04-25T08:36:13.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj545 td/osd-recovery-space/datafile 2024-04-25T08:36:13.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj546 td/osd-recovery-space/datafile 2024-04-25T08:36:13.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj547 td/osd-recovery-space/datafile 2024-04-25T08:36:13.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj548 td/osd-recovery-space/datafile 2024-04-25T08:36:13.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj549 td/osd-recovery-space/datafile 2024-04-25T08:36:13.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj550 td/osd-recovery-space/datafile 2024-04-25T08:36:13.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj551 td/osd-recovery-space/datafile 2024-04-25T08:36:13.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj552 td/osd-recovery-space/datafile 2024-04-25T08:36:13.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj553 td/osd-recovery-space/datafile 2024-04-25T08:36:13.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj554 td/osd-recovery-space/datafile 2024-04-25T08:36:13.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj555 td/osd-recovery-space/datafile 2024-04-25T08:36:13.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:13.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj556 td/osd-recovery-space/datafile 2024-04-25T08:36:14.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj557 td/osd-recovery-space/datafile 2024-04-25T08:36:14.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj558 td/osd-recovery-space/datafile 2024-04-25T08:36:14.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj559 td/osd-recovery-space/datafile 2024-04-25T08:36:14.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj560 td/osd-recovery-space/datafile 2024-04-25T08:36:14.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj561 td/osd-recovery-space/datafile 2024-04-25T08:36:14.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj562 td/osd-recovery-space/datafile 2024-04-25T08:36:14.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj563 td/osd-recovery-space/datafile 2024-04-25T08:36:14.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj564 td/osd-recovery-space/datafile 2024-04-25T08:36:14.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj565 td/osd-recovery-space/datafile 2024-04-25T08:36:14.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj566 td/osd-recovery-space/datafile 2024-04-25T08:36:14.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:14.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj567 td/osd-recovery-space/datafile 2024-04-25T08:36:15.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj568 td/osd-recovery-space/datafile 2024-04-25T08:36:15.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj569 td/osd-recovery-space/datafile 2024-04-25T08:36:15.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj570 td/osd-recovery-space/datafile 2024-04-25T08:36:15.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj571 td/osd-recovery-space/datafile 2024-04-25T08:36:15.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj572 td/osd-recovery-space/datafile 2024-04-25T08:36:15.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj573 td/osd-recovery-space/datafile 2024-04-25T08:36:15.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj574 td/osd-recovery-space/datafile 2024-04-25T08:36:15.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj575 td/osd-recovery-space/datafile 2024-04-25T08:36:15.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj576 td/osd-recovery-space/datafile 2024-04-25T08:36:15.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj577 td/osd-recovery-space/datafile 2024-04-25T08:36:15.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj578 td/osd-recovery-space/datafile 2024-04-25T08:36:15.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj579 td/osd-recovery-space/datafile 2024-04-25T08:36:15.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:15.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj580 td/osd-recovery-space/datafile 2024-04-25T08:36:16.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj581 td/osd-recovery-space/datafile 2024-04-25T08:36:16.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj582 td/osd-recovery-space/datafile 2024-04-25T08:36:16.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj583 td/osd-recovery-space/datafile 2024-04-25T08:36:16.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj584 td/osd-recovery-space/datafile 2024-04-25T08:36:16.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj585 td/osd-recovery-space/datafile 2024-04-25T08:36:16.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj586 td/osd-recovery-space/datafile 2024-04-25T08:36:16.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj587 td/osd-recovery-space/datafile 2024-04-25T08:36:16.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj588 td/osd-recovery-space/datafile 2024-04-25T08:36:16.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj589 td/osd-recovery-space/datafile 2024-04-25T08:36:16.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj590 td/osd-recovery-space/datafile 2024-04-25T08:36:16.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj591 td/osd-recovery-space/datafile 2024-04-25T08:36:16.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:16.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj592 td/osd-recovery-space/datafile 2024-04-25T08:36:17.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:17.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj593 td/osd-recovery-space/datafile 2024-04-25T08:36:17.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:17.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj594 td/osd-recovery-space/datafile 2024-04-25T08:36:17.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:17.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj595 td/osd-recovery-space/datafile 2024-04-25T08:36:17.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:17.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj596 td/osd-recovery-space/datafile 2024-04-25T08:36:17.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:17.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj597 td/osd-recovery-space/datafile 2024-04-25T08:36:17.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:17.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj598 td/osd-recovery-space/datafile 2024-04-25T08:36:17.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:17.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj599 td/osd-recovery-space/datafile 2024-04-25T08:36:17.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:115: TEST_recovery_test_simple: for o in $(seq 1 $objects) 2024-04-25T08:36:17.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:117: TEST_recovery_test_simple: rados -p test1 put obj600 td/osd-recovery-space/datafile 2024-04-25T08:36:17.780 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: expr 2 - 1 2024-04-25T08:36:17.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: seq 0 1 2024-04-25T08:36:17.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:36:17.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.0 injectargs '--fake_statfs_for_testing 3686400' 2024-04-25T08:36:17.936 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T08:36:17.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:120: TEST_recovery_test_simple: for o in $(seq 0 $(expr $OSDS - 1)) 2024-04-25T08:36:17.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:122: TEST_recovery_test_simple: ceph tell osd.1 injectargs '--fake_statfs_for_testing 3686400' 2024-04-25T08:36:18.087 INFO:tasks.workunit.client.0.smithi134.stdout:{} 2024-04-25T08:36:18.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:124: TEST_recovery_test_simple: sleep 5 2024-04-25T08:36:23.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:126: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:36:23.512 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:36:23.512 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 600 0 0 0 0 3072000 0 0 600 0 600 active+clean 2024-04-25T08:35:14.803627+0000 23'600 23:619 [1] 1 [1] 1 0'0 2024-04-25T08:35:12.192608+0000 0'0 2024-04-25T08:35:12.192608+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:08:16.567092+0000 0 0 2024-04-25T08:36:23.512 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:36:23.512 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:36:23.512 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:36:23.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:36:23.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:128: TEST_recovery_test_simple: for p in $(seq 1 $pools) 2024-04-25T08:36:23.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:130: TEST_recovery_test_simple: ceph osd pool set test1 size 2 2024-04-25T08:36:24.249 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 2 2024-04-25T08:36:24.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:134: TEST_recovery_test_simple: wait_for_recovery_toofull 30 2024-04-25T08:36:24.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:77: wait_for_recovery_toofull: local timeout=30 2024-04-25T08:36:24.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:78: wait_for_recovery_toofull: wait_for_state recovery_toofull 30 2024-04-25T08:36:24.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:52: wait_for_state: local state=recovery_toofull 2024-04-25T08:36:24.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:53: wait_for_state: local cur_in_state 2024-04-25T08:36:24.270 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: get_timeout_delays 30 5 2024-04-25T08:36:24.270 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:36:24.271 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:36:24.271 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:36:24.271 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:36:24.271 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:36:24.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: delays=('5' '10' '15') 2024-04-25T08:36:24.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:54: wait_for_state: local -a delays 2024-04-25T08:36:24.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:55: wait_for_state: local -i loop=0 2024-04-25T08:36:24.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:57: wait_for_state: flush_pg_stats 2024-04-25T08:36:24.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:36:24.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:36:24.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:36:24.789 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T08:36:24.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:36:24.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:24.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:36:24.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-04-25T08:36:24.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-04-25T08:36:24.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-04-25T08:36:24.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:36:24.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:36:25.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-04-25T08:36:25.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-04-25T08:36:25.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672985' 2024-04-25T08:36:25.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:25.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-04-25T08:36:25.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:25.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:36:25.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-04-25T08:36:25.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:25.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-04-25T08:36:25.241 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836510 2024-04-25T08:36:25.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-04-25T08:36:25.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:36:25.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836510 2024-04-25T08:36:25.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:36:26.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:36:26.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:36:27.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-04-25T08:36:27.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:36:27.196 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-04-25T08:36:27.196 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:36:27.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:36:27.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-04-25T08:36:27.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:36:27.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-04-25T08:36:27.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-25T08:36:27.200 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672985 2024-04-25T08:36:27.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:36:27.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-04-25T08:36:27.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: get_num_pgs 2024-04-25T08:36:27.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:36:27.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:36:28.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:58: wait_for_state: test 1 == 0 2024-04-25T08:36:28.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:62: wait_for_state: true 2024-04-25T08:36:28.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: get_num_in_state recovery_toofull 2024-04-25T08:36:28.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:43: get_num_in_state: local state=recovery_toofull 2024-04-25T08:36:28.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:44: get_num_in_state: local expression 2024-04-25T08:36:28.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:45: get_num_in_state: expression+='select(contains("recovery_toofull"))' 2024-04-25T08:36:28.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:46: get_num_in_state: ceph --format json pg dump pgs 2024-04-25T08:36:28.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:47: get_num_in_state: jq '.pg_stats | [.[] | .state | select(contains("recovery_toofull"))] | length' 2024-04-25T08:36:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:63: wait_for_state: cur_in_state=1 2024-04-25T08:36:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: test 1 -gt 0 2024-04-25T08:36:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:64: wait_for_state: break 2024-04-25T08:36:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:72: wait_for_state: return 0 2024-04-25T08:36:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:136: TEST_recovery_test_simple: ERRORS=0 2024-04-25T08:36:28.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:36:28.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: grep +recovery_toofull 2024-04-25T08:36:28.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: wc -l 2024-04-25T08:36:29.093 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:36:29.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:137: TEST_recovery_test_simple: '[' 1 '!=' 1 ']' 2024-04-25T08:36:29.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:143: TEST_recovery_test_simple: ceph pg dump pgs 2024-04-25T08:36:29.493 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T08:36:29.493 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 600 0 600 0 0 3072000 0 0 600 0 600 active+recovery_toofull+undersized+degraded+remapped 2024-04-25T08:36:25.053357+0000 23'600 27:630 [1,0] 1 [1] 1 0'0 2024-04-25T08:35:12.192608+0000 0'0 2024-04-25T08:35:12.192608+0000 0 0 no scrub is scheduled 0 0 2024-04-25T08:36:29.493 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:36:29.493 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T08:36:29.493 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T08:36:29.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:144: TEST_recovery_test_simple: ceph status 2024-04-25T08:36:30.043 INFO:tasks.workunit.client.0.smithi134.stdout: cluster: 2024-04-25T08:36:30.043 INFO:tasks.workunit.client.0.smithi134.stdout: id: 59c4ff4d-62f3-43b9-8fb7-3fa16eda36eb 2024-04-25T08:36:30.043 INFO:tasks.workunit.client.0.smithi134.stdout: health: HEALTH_ERR 2024-04-25T08:36:30.043 INFO:tasks.workunit.client.0.smithi134.stdout: 1 full osd(s) 2024-04-25T08:36:30.043 INFO:tasks.workunit.client.0.smithi134.stdout: Degraded data redundancy: 600/1200 objects degraded (50.000%), 1 pg degraded 2024-04-25T08:36:30.043 INFO:tasks.workunit.client.0.smithi134.stdout: Full OSDs blocking recovery: 1 pg recovery_toofull 2024-04-25T08:36:30.043 INFO:tasks.workunit.client.0.smithi134.stdout: 1 pool(s) do not have an application enabled 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: 1 pool(s) full 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: OSD count 2 < osd_pool_default_size 3 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: services: 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: mon: 1 daemons, quorum a (age 2m) 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: mgr: x(active, since 2m) 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: osd: 2 osds: 2 up (since 82s), 2 in (since 2m); 1 remapped pgs 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: data: 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: pools: 1 pools, 1 pgs 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: objects: 600 objects, 2.9 MiB 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: usage: 2.9 MiB used, 4.1 MiB / 7.0 MiB avail 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: pgs: 600/1200 objects degraded (50.000%) 2024-04-25T08:36:30.044 INFO:tasks.workunit.client.0.smithi134.stdout: 1 active+recovery_toofull+undersized+degraded+remapped 2024-04-25T08:36:30.045 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T08:36:30.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:145: TEST_recovery_test_simple: ceph status --format=json-pretty 2024-04-25T08:36:30.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.severity td/osd-recovery-space/stat.json 2024-04-25T08:36:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: eval 'SEV="HEALTH_ERR"' 2024-04-25T08:36:30.711 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:147: TEST_recovery_test_simple: SEV=HEALTH_ERR 2024-04-25T08:36:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:148: TEST_recovery_test_simple: '[' HEALTH_ERR '!=' HEALTH_ERR ']' 2024-04-25T08:36:30.711 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: jq .health.checks.PG_RECOVERY_FULL.summary.message td/osd-recovery-space/stat.json 2024-04-25T08:36:30.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: eval 'MSG="Full' OSDs blocking recovery: 1 pg 'recovery_toofull"' 2024-04-25T08:36:30.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:152: TEST_recovery_test_simple: MSG='Full OSDs blocking recovery: 1 pg recovery_toofull' 2024-04-25T08:36:30.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:153: TEST_recovery_test_simple: '[' 'Full OSDs blocking recovery: 1 pg recovery_toofull' '!=' 'Full OSDs blocking recovery: 1 pg recovery_toofull' ']' 2024-04-25T08:36:30.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:157: TEST_recovery_test_simple: rm -f td/osd-recovery-space/stat.json 2024-04-25T08:36:30.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:159: TEST_recovery_test_simple: '[' 0 '!=' 0 ']' 2024-04-25T08:36:30.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: seq 1 1 2024-04-25T08:36:30.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:164: TEST_recovery_test_simple: for i in $(seq 1 $pools) 2024-04-25T08:36:30.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:166: TEST_recovery_test_simple: delete_pool test1 2024-04-25T08:36:30.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test1 2024-04-25T08:36:30.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2024-04-25T08:36:31.256 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test1' does not exist 2024-04-25T08:36:31.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:168: TEST_recovery_test_simple: kill_daemons td/osd-recovery-space 2024-04-25T08:36:31.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:36:31.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:36:31.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:36:31.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:36:31.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:36:36.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:36:36.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-space.sh:37: run: teardown td/osd-recovery-space 2024-04-25T08:36:36.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:36:36.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:36:36.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:36:36.592 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:36:36.592 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:36:36.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:36:36.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:36:36.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:36:36.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:36:36.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:36:36.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:36:36.603 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:36:36.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:36:36.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:36:36.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:36:36.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:36:36.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:36:36.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:36:36.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:36:36.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:36:36.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:36:36.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:36:36.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:36:36.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:36:36.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:36:36.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.263450 2024-04-25T08:36:36.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:36:36.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:36:36.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T08:36:36.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-space 0 2024-04-25T08:36:36.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-space 2024-04-25T08:36:36.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T08:36:36.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-space KILL 2024-04-25T08:36:36.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:36:36.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:36:36.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:36:36.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:36:36.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:36:36.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:36:36.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:36:36.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:36:36.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:36:36.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:36:36.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:36:36.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:36:36.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:36:36.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:36:36.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:36:36.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:36:36.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:36:36.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T08:36:36.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-space 2024-04-25T08:36:36.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:36:36.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:36:36.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.263450 2024-04-25T08:36:36.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.263450 2024-04-25T08:36:36.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:36:36.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:36:36.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T08:36:36.659 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T08:36:36.659 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T08:36:36.673 INFO:tasks.workunit:Running workunit osd/osd-recovery-stats.sh... 2024-04-25T08:36:36.674 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-recovery-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh 2024-04-25T08:36:36.727 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-recovery-stats 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:21: run: local dir=td/osd-recovery-stats 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:22: run: shift 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T08:36:36.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:25: run: CEPH_MON=127.0.0.1:7115 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:26: run: export CEPH_ARGS 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: uuidgen 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:27: run: CEPH_ARGS+='--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none ' 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:30: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: export margin=10 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:31: run: margin=10 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: export objects=200 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:32: run: objects=200 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: export poolname=test 2024-04-25T08:36:36.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:33: run: poolname=test 2024-04-25T08:36:36.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: set 2024-04-25T08:36:36.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T08:36:36.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:35: run: local 'funcs=TEST_recovery_erasure_out1 2024-04-25T08:36:36.738 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_recovery_erasure_remapped 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_recovery_multi 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_recovery_replicated_out1 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_recovery_sizedown 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_recovery_sizeup 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_recovery_undersized' 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:36:36.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:36:36.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:36:36.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:36:36.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:36:36.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:36:36.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:36:36.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:36:36.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:36:36.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:36:36.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:36:36.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:36:36.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:36:36.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:36:36.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:36:36.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:36:36.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:36:36.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:36:36.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:36:36.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:36:36.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:36:36.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:36:36.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T08:36:36.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:36:36.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:36:36.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:36:36.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:36:36.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:36:36.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:36:36.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.280166 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_out1 td/osd-recovery-stats 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:173: TEST_recovery_erasure_out1: local dir=td/osd-recovery-stats 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:175: TEST_recovery_erasure_out1: do_recovery_out1 td/osd-recovery-stats erasure 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-04-25T08:36:36.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=erasure 2024-04-25T08:36:36.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-04-25T08:36:36.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:36:36.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:36:36.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:36:36.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:36:36.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:36:36.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:36:37.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:36:37.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:36:37.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:36:37.255 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:36:37.255 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:36:37.255 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:36:37.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:36:37.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:36:37.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:36:37.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:36:37.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:36:37.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:36:37.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:36:37.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:36:37.500 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:36:37.500 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:36:37.501 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:36:37.501 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:36:37.501 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:36:37.502 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:36:37.502 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:36:37.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:36:37.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get fsid 2024-04-25T08:36:37.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:36:37.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:36:37.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:36:37.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:36:37.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:36:37.621 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:36:37.621 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:36:37.621 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:36:37.622 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:36:37.622 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:36:37.622 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:36:37.622 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:36:37.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:36:37.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get mon_host 2024-04-25T08:36:37.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-04-25T08:36:37.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:36:37.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:36:37.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:36:37.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:36:37.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:36:37.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:36:38.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:36:38.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:36:38.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:36:38.647 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:36:38.647 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:36:38.647 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:36:38.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:36:38.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:36:38.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:36:38.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-04-25T08:36:38.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:36:38.685 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:36:38.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:36:38.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:36:38.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:36:38.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:36:38.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:36:38.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:36:38.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:36:38.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:36:38.688 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 14108ebe-9d89-41f1-bbf0-152c64d8e083 2024-04-25T08:36:38.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=14108ebe-9d89-41f1-bbf0-152c64d8e083 2024-04-25T08:36:38.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 14108ebe-9d89-41f1-bbf0-152c64d8e083' 2024-04-25T08:36:38.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:36:38.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWFipmQyF3KhAAa+iI6SHAk3mYaTo94DS8ug== 2024-04-25T08:36:38.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWFipmQyF3KhAAa+iI6SHAk3mYaTo94DS8ug=="}' 2024-04-25T08:36:38.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 14108ebe-9d89-41f1-bbf0-152c64d8e083 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:36:38.993 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:36:39.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:36:39.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWFipmQyF3KhAAa+iI6SHAk3mYaTo94DS8ug== --osd-uuid 14108ebe-9d89-41f1-bbf0-152c64d8e083 2024-04-25T08:36:39.050 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:39.057+0000 7f42cb9957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:39.053 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:39.061+0000 7f42cb9957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:39.055 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:39.061+0000 7f42cb9957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:39.055 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:39.061+0000 7f42cb9957c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:36:39.055 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:39.061+0000 7f42cb9957c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:36:39.055 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:39.061+0000 7f42cb9957c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:36:39.085 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:39.093+0000 7f42cb9957c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:36:45.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:36:45.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:36:45.491 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:36:45.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:36:45.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:36:46.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:36:46.052 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:36:46.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:36:46.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:36:46.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:36:46.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:36:46.083 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:46.089+0000 7f2b1030d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:46.085 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:46.093+0000 7f2b1030d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:46.087 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:46.093+0000 7f2b1030d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:36:46.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:36:46.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:36:46.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:36:46.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:36:46.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:36:46.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:46.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:36:46.464 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:36:46.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:46.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:36:46.670 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:46.677+0000 7f2b1030d7c0 -1 Falling back to public interface 2024-04-25T08:36:46.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:47.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:47.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:47.908 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:36:47.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:36:47.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:47.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:36:48.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:49.283 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:36:49.289+0000 7f2b1030d7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:36:49.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:49.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:49.313 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:36:49.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:36:49.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:49.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:36:49.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:50.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:50.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:50.985 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:36:50.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:36:50.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:50.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:36:51.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:52.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:52.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:52.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:36:52.459 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:36:52.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:52.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:36:52.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:53.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:53.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:53.885 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:36:53.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:36:53.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:53.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:36:54.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:55.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:55.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:55.414 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:36:55.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:36:55.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:55.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:36:55.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:56.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:56.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:56.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:36:56.843 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:36:56.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:56.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:36:57.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:58.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:58.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:58.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:36:58.448 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:36:58.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:58.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:36:58.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:36:59.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:36:59.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:36:59.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:36:59.885 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:36:59.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:36:59.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:00.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:01.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:01.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:01.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:37:01.441 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:37:01.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:01.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:01.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:37:02.897 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:37:02.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:02.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:03.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:04.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:04.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:04.930 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:37:04.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:37:04.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:04.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:06.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:07.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:07.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:07.079 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:37:07.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:37:07.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:07.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:07.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:08.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:08.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:08.969 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:37:08.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:37:08.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:08.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:09.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:10.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:10.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:10.798 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:37:10.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:37:10.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:10.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:11.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:12.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:12.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:12.944 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:37:12.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:37:12.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:12.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:13.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:14.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:14.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:14.549 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:37:14.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:37:14.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:14.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:37:14.949 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2767740724,v1:127.0.0.1:6803/2767740724] [v2:127.0.0.1:6804/2767740724,v1:127.0.0.1:6805/2767740724] exists,up 14108ebe-9d89-41f1-bbf0-152c64d8e083 2024-04-25T08:37:14.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:37:14.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:37:14.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:37:14.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:37:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ecbf34b0-9970-4d7c-b00d-74be5a77b350 2024-04-25T08:37:14.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ecbf34b0-9970-4d7c-b00d-74be5a77b350' 2024-04-25T08:37:14.956 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 ecbf34b0-9970-4d7c-b00d-74be5a77b350 2024-04-25T08:37:14.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:37:14.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA6FipmH4JaOhAAvSMiHSx3pHlAWDhqShl6jA== 2024-04-25T08:37:14.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA6FipmH4JaOhAAvSMiHSx3pHlAWDhqShl6jA=="}' 2024-04-25T08:37:14.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ecbf34b0-9970-4d7c-b00d-74be5a77b350 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:37:15.556 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:37:15.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:37:15.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA6FipmH4JaOhAAvSMiHSx3pHlAWDhqShl6jA== --osd-uuid ecbf34b0-9970-4d7c-b00d-74be5a77b350 2024-04-25T08:37:15.606 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:15.612+0000 7f1dc20b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:15.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:15.644+0000 7f1dc20b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:15.640 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:15.644+0000 7f1dc20b87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:15.640 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:15.648+0000 7f1dc20b87c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:37:15.641 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:15.648+0000 7f1dc20b87c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:37:15.641 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:15.648+0000 7f1dc20b87c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:37:15.678 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:15.684+0000 7f1dc20b87c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:37:22.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:37:22.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:37:22.173 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:37:22.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:37:22.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:37:22.734 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:37:22.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:37:22.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:37:22.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:37:22.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:37:22.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:37:22.765 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:22.772+0000 7f21f32187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:22.767 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:22.772+0000 7f21f32187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:22.769 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:22.776+0000 7f21f32187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:23.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:37:23.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:37:23.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:37:23.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:37:23.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:37:23.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:23.179 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:37:23.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:37:23.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:23.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:23.306 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:23.312+0000 7f21f32187c0 -1 Falling back to public interface 2024-04-25T08:37:23.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:24.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:24.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:24.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:37:24.641 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:37:24.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:24.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:25.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:26.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:26.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:26.045 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:37:26.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:37:26.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:26.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:26.095 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:26.100+0000 7f21f32187c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:37:26.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:27.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:27.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:27.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:37:27.633 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:37:27.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:27.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:28.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:29.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:29.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:29.055 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:37:29.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:37:29.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:29.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:29.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:30.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:30.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:30.496 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:37:30.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:37:30.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:30.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:30.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:31.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:31.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:31.977 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:37:31.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:37:31.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:31.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:32.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:33.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:33.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:33.425 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:37:33.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:37:33.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:33.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:33.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:34.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:34.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:34.944 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:37:34.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:37:34.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:34.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:35.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:36.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:36.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:36.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:37:36.394 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:37:36.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:36.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:36.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:37.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:37.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:37.978 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:37:37.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:37:37.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:37.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:38.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:39.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:39.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:39.461 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:37:39.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:37:39.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:39.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:39.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:40.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:40.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:40.885 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:37:40.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:37:40.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:40.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:41.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:42.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:42.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:42.431 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:37:42.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:37:42.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:42.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:43.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:44.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:44.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:44.303 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:37:44.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:37:44.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:44.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:45.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:46.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:46.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:46.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:37:46.149 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:37:46.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:46.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:46.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:47.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:47.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:47.948 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:37:47.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:37:47.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:47.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:48.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:49.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:49.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:49.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:37:49.542 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:37:49.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:49.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:50.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:37:51.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:37:51.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:37:51.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:37:51.641 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:37:51.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:37:51.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:37:52.083 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2403817090,v1:127.0.0.1:6811/2403817090] [v2:127.0.0.1:6812/2403817090,v1:127.0.0.1:6813/2403817090] exists,up ecbf34b0-9970-4d7c-b00d-74be5a77b350 2024-04-25T08:37:52.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:37:52.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:37:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:37:52.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:37:52.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:37:52.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:37:52.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:37:52.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:37:52.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:37:52.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc53f500-4974-4059-868b-31bdc74f1691 2024-04-25T08:37:52.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cc53f500-4974-4059-868b-31bdc74f1691' 2024-04-25T08:37:52.090 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 cc53f500-4974-4059-868b-31bdc74f1691 2024-04-25T08:37:52.090 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:37:52.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBgFipmNi/MBhAAc6tcujlK8wI0mxzSQO3riw== 2024-04-25T08:37:52.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBgFipmNi/MBhAAc6tcujlK8wI0mxzSQO3riw=="}' 2024-04-25T08:37:52.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc53f500-4974-4059-868b-31bdc74f1691 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:37:52.762 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:37:52.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:37:52.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBgFipmNi/MBhAAc6tcujlK8wI0mxzSQO3riw== --osd-uuid cc53f500-4974-4059-868b-31bdc74f1691 2024-04-25T08:37:52.814 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:52.819+0000 7f592fe617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:52.847 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:52.851+0000 7f592fe617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:52.848 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:52.851+0000 7f592fe617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:37:52.849 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:52.855+0000 7f592fe617c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:37:52.849 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:52.855+0000 7f592fe617c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:37:52.849 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:52.855+0000 7f592fe617c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:37:52.886 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:37:52.891+0000 7f592fe617c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:37:59.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:37:59.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:37:59.663 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T08:37:59.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:37:59.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:38:00.207 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T08:38:00.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:38:00.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:38:00.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:38:00.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:38:00.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:38:00.233 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:00.239+0000 7f0deda9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:00.235 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:00.239+0000 7f0deda9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:00.236 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:00.243+0000 7f0deda9b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:00.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:38:00.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:38:00.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:38:00.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:38:00.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:38:00.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:00.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:38:00.628 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:38:00.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:00.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:00.778 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:00.783+0000 7f0deda9b7c0 -1 Falling back to public interface 2024-04-25T08:38:01.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:02.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:02.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:02.084 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:38:02.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:38:02.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:02.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:02.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:03.456 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:03.459+0000 7f0deda9b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:38:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:38:03.501 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:38:03.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:03.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:03.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:04.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:04.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:04.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:38:04.982 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:38:04.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:04.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:05.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:06.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:06.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:06.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:38:06.570 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:38:06.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:06.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:07.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:08.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:08.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:08.013 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:38:08.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:38:08.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:08.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:08.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:09.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:09.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:09.507 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:38:09.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:38:09.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:09.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:09.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:10.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:10.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:10.921 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:38:10.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:38:10.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:10.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:11.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:12.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:12.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:12.552 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:38:12.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:38:12.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:12.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:12.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:13.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:13.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:13.985 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:38:13.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:38:13.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:13.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:14.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:15.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:15.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:15.512 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:38:15.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:38:15.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:15.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:16.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:17.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:17.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:17.019 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:38:17.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:38:17.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:17.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:17.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:18.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:18.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:18.604 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:38:18.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:38:18.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:18.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:19.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:20.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:20.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:20.176 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:38:20.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:38:20.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:20.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:20.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:21.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:21.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:21.710 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:38:21.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:38:21.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:21.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:22.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:23.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:23.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:23.830 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:38:23.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:38:23.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:23.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:24.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:25.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:25.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:25.637 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:38:25.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:38:25.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:25.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:26.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:27.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:27.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:27.326 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:38:27.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:38:27.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:27.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:28.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:29.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:29.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:29.099 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:38:29.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:38:29.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:29.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:38:29.575 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1808393654,v1:127.0.0.1:6819/1808393654] [v2:127.0.0.1:6820/1808393654,v1:127.0.0.1:6821/1808393654] exists,up cc53f500-4974-4059-868b-31bdc74f1691 2024-04-25T08:38:29.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:38:29.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:38:29.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:38:29.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:38:29.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:38:29.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:38:29.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:38:29.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:38:29.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:38:29.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:38:29.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:38:29.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:38:29.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:38:29.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b5502c55-2eca-411a-8192-70e32d061bac 2024-04-25T08:38:29.581 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 b5502c55-2eca-411a-8192-70e32d061bac 2024-04-25T08:38:29.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 b5502c55-2eca-411a-8192-70e32d061bac' 2024-04-25T08:38:29.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:38:29.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCFFipmxK58JBAAfqTb+GEU8GIDJHcTj2dThA== 2024-04-25T08:38:29.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCFFipmxK58JBAAfqTb+GEU8GIDJHcTj2dThA=="}' 2024-04-25T08:38:29.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b5502c55-2eca-411a-8192-70e32d061bac -i td/osd-recovery-stats/3/new.json 2024-04-25T08:38:30.101 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:38:30.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:38:30.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCFFipmxK58JBAAfqTb+GEU8GIDJHcTj2dThA== --osd-uuid b5502c55-2eca-411a-8192-70e32d061bac 2024-04-25T08:38:30.153 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:30.158+0000 7fcab838f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:30.179 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:30.182+0000 7fcab838f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:30.180 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:30.186+0000 7fcab838f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:30.181 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:30.186+0000 7fcab838f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:38:30.181 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:30.186+0000 7fcab838f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:38:30.181 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:30.186+0000 7fcab838f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:38:30.218 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:30.222+0000 7fcab838f7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:38:37.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:38:37.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:38:37.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:38:37.382 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T08:38:37.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:38:37.994 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T08:38:37.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:38:37.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:38:37.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:38:37.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:38:37.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:38:38.026 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:38.030+0000 7f95c67c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:38.047 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:38.050+0000 7f95c67c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:38.048 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:38.054+0000 7f95c67c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:38:38.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:38:38.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:38:38.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:38:38.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:38:38.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:38:38.413 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:38:38.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:38.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:38:38.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:38.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:38.594 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:38.598+0000 7f95c67c97c0 -1 Falling back to public interface 2024-04-25T08:38:38.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:39.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:39.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:39.837 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:38:39.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:38:39.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:39.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:40.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:41.237 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:38:41.242+0000 7f95c67c97c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:38:41.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:41.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:41.244 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:38:41.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:38:41.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:41.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:41.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:42.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:42.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:42.751 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:38:42.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:38:42.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:42.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:43.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:44.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:44.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:44.242 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:38:44.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:38:44.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:44.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:44.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:45.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:45.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:45.660 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:38:45.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:38:45.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:45.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:46.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:47.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:47.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:47.213 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:38:47.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:38:47.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:47.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:47.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:48.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:48.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:48.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:38:48.767 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:38:48.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:48.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:49.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:50.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:50.229 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:38:50.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:50.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:38:50.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:50.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:50.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:51.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:51.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:51.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:38:51.654 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:38:51.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:51.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:52.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:53.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:53.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:53.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:38:53.209 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:38:53.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:53.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:53.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:54.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:54.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:54.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:38:54.724 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:38:54.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:54.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:55.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:56.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:56.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:56.528 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:38:56.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:38:56.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:56.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:57.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:38:58.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:38:58.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:38:58.606 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:38:58.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:38:58.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:38:58.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:38:59.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:00.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:00.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:00.490 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:39:00.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:39:00.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:00.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:01.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:02.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:02.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:02.364 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:39:02.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:39:02.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:02.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:03.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:04.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:04.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:04.361 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:39:04.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:39:04.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:04.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:05.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:06.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:06.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:06.056 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:39:06.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:39:06.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:06.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:06.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:07.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:07.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:07.523 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:39:07.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:39:07.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:07.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:39:07.978 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4214225262,v1:127.0.0.1:6827/4214225262] [v2:127.0.0.1:6828/4214225262,v1:127.0.0.1:6829/4214225262] exists,up b5502c55-2eca-411a-8192-70e32d061bac 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:39:07.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:39:07.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:39:07.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:39:07.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:39:07.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:39:07.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:39:07.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:39:07.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:39:07.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:39:07.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:39:07.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:39:07.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:39:07.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:39:07.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:39:07.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:39:07.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:39:07.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9dfcf545-767d-4650-8fdd-a4253ace4fcc 2024-04-25T08:39:07.985 INFO:tasks.workunit.client.0.smithi134.stdout:add osd4 9dfcf545-767d-4650-8fdd-a4253ace4fcc 2024-04-25T08:39:07.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 9dfcf545-767d-4650-8fdd-a4253ace4fcc' 2024-04-25T08:39:07.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:39:08.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCsFipmOyifABAAQB3Cla3ivEVgDvjTc9xzqA== 2024-04-25T08:39:08.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCsFipmOyifABAAQB3Cla3ivEVgDvjTc9xzqA=="}' 2024-04-25T08:39:08.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9dfcf545-767d-4650-8fdd-a4253ace4fcc -i td/osd-recovery-stats/4/new.json 2024-04-25T08:39:08.511 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:39:08.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:39:08.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCsFipmOyifABAAQB3Cla3ivEVgDvjTc9xzqA== --osd-uuid 9dfcf545-767d-4650-8fdd-a4253ace4fcc 2024-04-25T08:39:08.557 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:08.566+0000 7fcabfe007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:08.571 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:08.578+0000 7fcabfe007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:08.573 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:08.582+0000 7fcabfe007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:08.573 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:08.582+0000 7fcabfe007c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:39:08.573 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:08.582+0000 7fcabfe007c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:39:08.573 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:08.582+0000 7fcabfe007c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:39:08.611 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:08.618+0000 7fcabfe007c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:39:15.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:39:15.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:39:15.363 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd4 key to auth repository 2024-04-25T08:39:15.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:39:15.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:39:15.917 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.4 2024-04-25T08:39:15.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:39:15.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:39:15.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:39:15.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:39:15.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:39:15.949 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:15.957+0000 7f97dd17c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:15.987 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:15.993+0000 7f97dd17c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:15.989 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:15.997+0000 7f97dd17c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:16.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:39:16.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:39:16.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:39:16.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:39:16.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:39:16.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:16.307 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:39:16.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:39:16.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:16.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:16.554 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:16.561+0000 7f97dd17c7c0 -1 Falling back to public interface 2024-04-25T08:39:16.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:17.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:17.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:17.753 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:39:17.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:39:17.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:17.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:18.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:19.169 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:39:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:39:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:19.178 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:19.185+0000 7f97dd17c7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:39:19.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:20.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:20.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:20.797 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:39:20.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:39:20.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:20.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:21.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:22.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:22.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:22.298 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:39:22.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:39:22.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:22.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:22.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:23.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:23.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:23.814 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:39:23.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:39:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:24.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:25.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:25.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:25.436 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:39:25.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:39:25.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:25.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:25.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:26.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:26.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:26.860 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:39:26.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:39:26.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:26.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:27.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:28.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:28.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:28.289 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:39:28.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:39:28.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:28.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:28.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:29.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:29.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:29.720 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:39:29.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:39:29.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:29.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:30.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:31.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:31.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:31.231 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:39:31.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:39:31.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:31.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:31.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:32.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:32.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:32.775 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:39:32.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:39:32.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:32.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:33.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:34.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:34.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:34.238 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:39:34.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:39:34.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:34.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:35.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:36.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:36.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:36.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:39:36.213 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:39:36.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:36.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:37.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:38.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:38.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:38.217 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:39:38.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:39:38.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:38.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:39.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:40.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:40.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:40.053 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:39:40.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:39:40.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:40.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:41.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:42.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:42.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:42.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:39:42.308 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:39:42.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:42.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:43.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:39:44.249 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:39:44.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:44.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:44.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:45.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:45.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:45.892 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:39:45.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:39:45.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:45.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:39:46.398 INFO:tasks.workunit.client.0.smithi134.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1042134613,v1:127.0.0.1:6835/1042134613] [v2:127.0.0.1:6836/1042134613,v1:127.0.0.1:6837/1042134613] exists,up 9dfcf545-767d-4650-8fdd-a4253ace4fcc 2024-04-25T08:39:46.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:39:46.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:39:46.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:39:46.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-04-25T08:39:46.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:39:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:39:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:39:46.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:39:46.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:39:46.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:39:46.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:39:46.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:39:46.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:39:46.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:39:46.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:39:46.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:39:46.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7cd06f0b-1b80-4318-af06-66cbb8e0e507 2024-04-25T08:39:46.405 INFO:tasks.workunit.client.0.smithi134.stdout:add osd5 7cd06f0b-1b80-4318-af06-66cbb8e0e507 2024-04-25T08:39:46.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 7cd06f0b-1b80-4318-af06-66cbb8e0e507' 2024-04-25T08:39:46.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:39:46.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSFipmI7W7GRAA/SVXs3V9Aq+Sfgrxn3IAbQ== 2024-04-25T08:39:46.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSFipmI7W7GRAA/SVXs3V9Aq+Sfgrxn3IAbQ=="}' 2024-04-25T08:39:46.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7cd06f0b-1b80-4318-af06-66cbb8e0e507 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:39:46.993 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:39:47.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:39:47.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSFipmI7W7GRAA/SVXs3V9Aq+Sfgrxn3IAbQ== --osd-uuid 7cd06f0b-1b80-4318-af06-66cbb8e0e507 2024-04-25T08:39:47.049 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:47.057+0000 7f291d9597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:47.051 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:47.057+0000 7f291d9597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:47.053 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:47.061+0000 7f291d9597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:47.053 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:47.061+0000 7f291d9597c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:39:47.053 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:47.061+0000 7f291d9597c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:39:47.053 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:47.061+0000 7f291d9597c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:39:47.132 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:47.141+0000 7f291d9597c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:39:53.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:39:53.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:39:53.836 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd5 key to auth repository 2024-04-25T08:39:53.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:39:53.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:39:54.410 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.5 2024-04-25T08:39:54.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:39:54.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:39:54.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:39:54.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:39:54.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:39:54.442 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:54.448+0000 7f19480807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:54.445 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:54.452+0000 7f19480807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:54.446 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:54.452+0000 7f19480807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:39:54.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:39:54.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:39:54.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:39:54.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:39:54.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:39:54.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:54.829 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:39:54.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:39:54.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:54.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:39:55.001 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:55.008+0000 7f19480807c0 -1 Falling back to public interface 2024-04-25T08:39:55.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:56.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:56.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:56.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:39:56.277 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:39:56.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:56.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:39:56.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:57.683 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:39:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:39:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:39:57.705 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:39:57.712+0000 7f19480807c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:39:58.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:39:59.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:39:59.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:39:59.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:39:59.267 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:39:59.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:39:59.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:39:59.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:00.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:00.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:00.892 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:40:00.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:40:00.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:00.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:01.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:02.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:02.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:02.344 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:40:02.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:40:02.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:02.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:02.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:03.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:03.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:03.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:40:03.996 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:40:03.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:03.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:04.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:05.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:05.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:05.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:40:05.481 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:40:05.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:05.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:06.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:07.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:07.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:07.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:40:07.070 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:40:07.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:07.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:07.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:08.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:08.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:08.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:40:08.719 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:40:08.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:08.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:09.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:10.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:10.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:10.167 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:40:10.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:40:10.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:10.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:10.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:11.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:11.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:11.750 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:40:11.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:40:11.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:11.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:12.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:13.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:13.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:13.403 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:40:13.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:40:13.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:13.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:13.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:14.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:14.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:40:14.976 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:40:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:14.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:16.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:17.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:17.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:17.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:40:17.034 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:40:17.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:17.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:17.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:18.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:18.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:18.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:40:18.852 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:40:18.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:18.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:19.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:20.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:20.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:20.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:40:20.748 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:40:20.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:20.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:21.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:22.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:22.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:22.477 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:40:22.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:40:22.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:22.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:23.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:40:24.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:40:24.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:40:24.298 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:40:24.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:40:24.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:40:24.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:40:24.718 INFO:tasks.workunit.client.0.smithi134.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3586132418,v1:127.0.0.1:6843/3586132418] [v2:127.0.0.1:6844/3586132418,v1:127.0.0.1:6845/3586132418] exists,up 7cd06f0b-1b80-4318-af06-66cbb8e0e507 2024-04-25T08:40:24.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:40:24.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:40:24.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:40:24.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' erasure = erasure ']' 2024-04-25T08:40:24.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:132: do_recovery_out1: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T08:40:25.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:133: do_recovery_out1: create_pool test 1 1 erasure myprofile 2024-04-25T08:40:25.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T08:40:26.390 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T08:40:26.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:40:27.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-04-25T08:40:27.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:40:27.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:40:27.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:40:27.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:40:27.412 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:40:27.412 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:40:27.412 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:40:27.412 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:40:27.413 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:40:27.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:40:27.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:40:27.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:40:27.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:40:27.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:40:27.539 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:40:28.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:40:28.104 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:40:28.104 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:40:28.104 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:40:28.105 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T08:40:28.105 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T08:40:28.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:40:28.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:40:28.105 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:40:28.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836524 2024-04-25T08:40:28.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836524 2024-04-25T08:40:28.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524' 2024-04-25T08:40:28.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:40:28.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:40:28.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672998 2024-04-25T08:40:28.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672998 2024-04-25T08:40:28.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998' 2024-04-25T08:40:28.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:40:28.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:40:28.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509471 2024-04-25T08:40:28.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509471 2024-04-25T08:40:28.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509471' 2024-04-25T08:40:28.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:40:28.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:40:28.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345952 2024-04-25T08:40:28.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345952 2024-04-25T08:40:28.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509471 3-85899345952' 2024-04-25T08:40:28.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:40:28.726 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:40:28.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182418 2024-04-25T08:40:28.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182418 2024-04-25T08:40:28.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509471 3-85899345952 4-107374182418' 2024-04-25T08:40:28.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:40:28.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:40:29.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018889 2024-04-25T08:40:29.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018889 2024-04-25T08:40:29.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836524 1-42949672998 2-64424509471 3-85899345952 4-107374182418 5-128849018889' 2024-04-25T08:40:29.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:40:29.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836524 2024-04-25T08:40:29.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:40:29.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:40:29.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836524 2024-04-25T08:40:29.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:40:29.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836524 2024-04-25T08:40:29.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-04-25T08:40:29.038 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836524 2024-04-25T08:40:29.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:40:29.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836524 -lt 21474836524 2024-04-25T08:40:29.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:40:29.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672998 2024-04-25T08:40:29.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:40:29.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:40:29.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:40:29.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672998 2024-04-25T08:40:29.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672998 2024-04-25T08:40:29.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672998' 2024-04-25T08:40:29.476 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672998 2024-04-25T08:40:29.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:40:29.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672998 -lt 42949672998 2024-04-25T08:40:29.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:40:29.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509471 2024-04-25T08:40:29.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:40:29.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:40:29.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:40:29.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509471 2024-04-25T08:40:29.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509471 2024-04-25T08:40:29.904 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509471 2024-04-25T08:40:29.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-04-25T08:40:29.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:40:30.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509471 -lt 64424509471 2024-04-25T08:40:30.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:40:30.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345952 2024-04-25T08:40:30.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:40:30.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:40:30.305 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345952 2024-04-25T08:40:30.305 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:40:30.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345952 2024-04-25T08:40:30.306 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345952 2024-04-25T08:40:30.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345952' 2024-04-25T08:40:30.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:40:30.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345952 -lt 85899345952 2024-04-25T08:40:30.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:40:30.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182418 2024-04-25T08:40:30.722 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:40:30.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:40:30.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182418 2024-04-25T08:40:30.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:40:30.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182418 2024-04-25T08:40:30.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182418' 2024-04-25T08:40:30.726 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182418 2024-04-25T08:40:30.726 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:40:31.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182418 -lt 107374182418 2024-04-25T08:40:31.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:40:31.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018889 2024-04-25T08:40:31.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:40:31.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:40:31.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018889 2024-04-25T08:40:31.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:40:31.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018889 2024-04-25T08:40:31.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018889' 2024-04-25T08:40:31.175 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018889 2024-04-25T08:40:31.175 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:40:31.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018890 -lt 128849018889 2024-04-25T08:40:31.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:40:31.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:40:31.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:40:32.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:40:32.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:40:32.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:40:32.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:40:32.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:40:32.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:40:32.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:40:32.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:40:32.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:40:32.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:40:32.706 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:40:32.706 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:40:33.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:40:33.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:40:33.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:40:33.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-04-25T08:40:33.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:33.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-04-25T08:40:33.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:33.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-04-25T08:40:33.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:33.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-04-25T08:40:34.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:34.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-04-25T08:40:34.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:34.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-04-25T08:40:34.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:34.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-04-25T08:40:34.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:34.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-04-25T08:40:34.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:34.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-04-25T08:40:35.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:35.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-04-25T08:40:35.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:35.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-04-25T08:40:35.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:35.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-04-25T08:40:35.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:35.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-04-25T08:40:35.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:35.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-04-25T08:40:36.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:36.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-04-25T08:40:36.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:36.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-04-25T08:40:36.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:36.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-04-25T08:40:36.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:36.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-04-25T08:40:36.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:36.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-04-25T08:40:36.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:36.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-04-25T08:40:37.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:37.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-04-25T08:40:37.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:37.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-04-25T08:40:37.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:37.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-04-25T08:40:37.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:37.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-04-25T08:40:37.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:37.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-04-25T08:40:38.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:38.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-04-25T08:40:38.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:38.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-04-25T08:40:38.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:38.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-04-25T08:40:38.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:38.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-04-25T08:40:38.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:38.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-04-25T08:40:39.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:39.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-04-25T08:40:39.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:39.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-04-25T08:40:39.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:39.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-04-25T08:40:39.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:39.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-04-25T08:40:39.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:39.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-04-25T08:40:40.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:40.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-04-25T08:40:40.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:40.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-04-25T08:40:40.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:40.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-04-25T08:40:40.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:40.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-04-25T08:40:40.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:40.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-04-25T08:40:41.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:41.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-04-25T08:40:41.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:41.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-04-25T08:40:41.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:41.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-04-25T08:40:41.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:41.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-04-25T08:40:41.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:41.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-04-25T08:40:42.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:42.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-04-25T08:40:42.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:42.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-04-25T08:40:42.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:42.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-04-25T08:40:42.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:42.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-04-25T08:40:42.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:42.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-04-25T08:40:42.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:42.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-04-25T08:40:43.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:43.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-04-25T08:40:43.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:43.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-04-25T08:40:43.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:43.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-04-25T08:40:43.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:43.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-04-25T08:40:44.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:44.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-04-25T08:40:44.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:44.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-04-25T08:40:44.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:44.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-04-25T08:40:44.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:44.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-04-25T08:40:44.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:44.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-04-25T08:40:44.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:44.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-04-25T08:40:45.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:45.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-04-25T08:40:45.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:45.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-04-25T08:40:45.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:45.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-04-25T08:40:45.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:45.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-04-25T08:40:45.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:45.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-04-25T08:40:46.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:46.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-04-25T08:40:46.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:46.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-04-25T08:40:46.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:46.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-04-25T08:40:46.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:46.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-04-25T08:40:47.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:47.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-04-25T08:40:47.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:47.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-04-25T08:40:47.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:47.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-04-25T08:40:47.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:47.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-04-25T08:40:47.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:47.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-04-25T08:40:48.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:48.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-04-25T08:40:48.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:48.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-04-25T08:40:48.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:48.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-04-25T08:40:48.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:48.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-04-25T08:40:48.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:48.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-04-25T08:40:49.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:49.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-04-25T08:40:49.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:49.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-04-25T08:40:49.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:49.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-04-25T08:40:49.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:49.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-04-25T08:40:49.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:49.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-04-25T08:40:50.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:50.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-04-25T08:40:50.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:50.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-04-25T08:40:50.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:50.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-04-25T08:40:50.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:50.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-04-25T08:40:51.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:51.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-04-25T08:40:51.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:51.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-04-25T08:40:51.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:51.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-04-25T08:40:51.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:51.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-04-25T08:40:51.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:51.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-04-25T08:40:52.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:52.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-04-25T08:40:52.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:52.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-04-25T08:40:52.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:52.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-04-25T08:40:52.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:52.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-04-25T08:40:52.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:52.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-04-25T08:40:52.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:52.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-04-25T08:40:53.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:53.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-04-25T08:40:53.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:53.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-04-25T08:40:53.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:53.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-04-25T08:40:53.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:53.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-04-25T08:40:54.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:54.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-04-25T08:40:54.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:54.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-04-25T08:40:54.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:54.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-04-25T08:40:54.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:54.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-04-25T08:40:54.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:54.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-04-25T08:40:55.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:55.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-04-25T08:40:55.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:55.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-04-25T08:40:55.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:55.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-04-25T08:40:55.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:55.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-04-25T08:40:55.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:55.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-04-25T08:40:56.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:56.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-04-25T08:40:56.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:56.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-04-25T08:40:56.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:56.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-04-25T08:40:56.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:56.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-04-25T08:40:56.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-04-25T08:40:57.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:57.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-04-25T08:40:57.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:57.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-04-25T08:40:57.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:57.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-04-25T08:40:57.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:57.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-04-25T08:40:57.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:57.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-04-25T08:40:58.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:58.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-04-25T08:40:58.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:58.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-04-25T08:40:58.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:58.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-04-25T08:40:58.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:58.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-04-25T08:40:58.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:58.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-04-25T08:40:59.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:59.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-04-25T08:40:59.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:59.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-04-25T08:40:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-04-25T08:40:59.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:59.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-04-25T08:40:59.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:40:59.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-04-25T08:41:00.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:00.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-04-25T08:41:00.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:00.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-04-25T08:41:00.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:00.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-04-25T08:41:00.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:00.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-04-25T08:41:00.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:00.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-04-25T08:41:01.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:01.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-04-25T08:41:01.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:01.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-04-25T08:41:01.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:01.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-04-25T08:41:01.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:01.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-04-25T08:41:01.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:01.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-04-25T08:41:02.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:02.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-04-25T08:41:02.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:02.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-04-25T08:41:02.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:02.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-04-25T08:41:02.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:02.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-04-25T08:41:02.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:02.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-04-25T08:41:03.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:03.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-04-25T08:41:03.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:03.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-04-25T08:41:03.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:03.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-04-25T08:41:03.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:03.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-04-25T08:41:03.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:03.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-04-25T08:41:04.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:04.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-04-25T08:41:04.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:04.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-04-25T08:41:04.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:04.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-04-25T08:41:04.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:04.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-04-25T08:41:05.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:05.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-04-25T08:41:05.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:05.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-04-25T08:41:05.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:05.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-04-25T08:41:05.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:05.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-04-25T08:41:05.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:05.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-04-25T08:41:06.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:06.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-04-25T08:41:06.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:06.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-04-25T08:41:06.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:06.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-04-25T08:41:06.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:06.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-04-25T08:41:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-04-25T08:41:07.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:07.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-04-25T08:41:07.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:07.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-04-25T08:41:07.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:07.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-04-25T08:41:07.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:07.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-04-25T08:41:07.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:07.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-04-25T08:41:08.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:08.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-04-25T08:41:08.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:08.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-04-25T08:41:08.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:08.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-04-25T08:41:08.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:08.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-04-25T08:41:08.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:08.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-04-25T08:41:09.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:09.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-04-25T08:41:09.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:09.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-04-25T08:41:09.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:09.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-04-25T08:41:09.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:09.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-04-25T08:41:09.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:09.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-04-25T08:41:10.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:10.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-04-25T08:41:10.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:10.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-04-25T08:41:10.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:10.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-04-25T08:41:10.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:10.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-04-25T08:41:10.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:10.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-04-25T08:41:11.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:11.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-04-25T08:41:11.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:11.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-04-25T08:41:11.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:11.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-04-25T08:41:11.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:11.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-04-25T08:41:11.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:11.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-04-25T08:41:12.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:12.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-04-25T08:41:12.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:12.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-04-25T08:41:12.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:12.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-04-25T08:41:12.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:12.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-04-25T08:41:12.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:12.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-04-25T08:41:13.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:13.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-04-25T08:41:13.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:13.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-04-25T08:41:13.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:41:13.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-04-25T08:41:13.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-04-25T08:41:13.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:41:13.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:41:13.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:41:13.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:41:14.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-04-25T08:41:14.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-04-25T08:41:14.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:41:14.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:41:14.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:41:14.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:41:14.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-04-25T08:41:14.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-04-25T08:41:14.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:41:14.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:41:14.574 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:41:14.575 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:41:14.575 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:41:14.575 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:41:14.575 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:41:14.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:41:14.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:41:14.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:41:15.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-04-25T08:41:15.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-04-25T08:41:16.217 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is set 2024-04-25T08:41:16.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:41:16.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 281247 2024-04-25T08:41:16.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-04-25T08:41:16.837 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T08:41:16.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-04-25T08:41:17.724 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T08:41:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-04-25T08:41:18.922 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is unset 2024-04-25T08:41:18.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-04-25T08:41:18.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:41:18.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:41:18.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:41:18.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:41:19.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:41:19.562 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:41:19.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-04-25T08:41:21.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-04-25T08:41:21.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:41:21.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:41:21.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:41:21.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:41:21.579 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:41:21.580 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:41:21.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:41:21.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:41:21.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:41:21.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:41:21.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:41:21.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:41:21.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:41:21.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:41:21.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:41:22.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:41:22.150 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:41:22.151 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:41:22.151 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:41:22.151 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T08:41:22.151 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T08:41:22.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:41:22.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:22.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:41:22.281 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:41:22.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:41:22.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:41:22.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:41:22.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:22.284 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:41:22.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673011 2024-04-25T08:41:22.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673011 2024-04-25T08:41:22.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011' 2024-04-25T08:41:22.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:22.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:41:22.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509484 2024-04-25T08:41:22.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509484 2024-04-25T08:41:22.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011 2-64424509484' 2024-04-25T08:41:22.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:22.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:41:22.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345965 2024-04-25T08:41:22.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345965 2024-04-25T08:41:22.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011 2-64424509484 3-85899345965' 2024-04-25T08:41:22.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:22.734 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:41:22.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182431 2024-04-25T08:41:22.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182431 2024-04-25T08:41:22.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011 2-64424509484 3-85899345965 4-107374182431' 2024-04-25T08:41:22.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:41:22.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:41:23.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018902 2024-04-25T08:41:23.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018902 2024-04-25T08:41:23.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673011 2-64424509484 3-85899345965 4-107374182431 5-128849018902' 2024-04-25T08:41:23.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:23.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673011 2024-04-25T08:41:23.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:23.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:41:23.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673011 2024-04-25T08:41:23.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:23.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673011 2024-04-25T08:41:23.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673011' 2024-04-25T08:41:23.045 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673011 2024-04-25T08:41:23.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:41:23.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673011 -lt 42949673011 2024-04-25T08:41:23.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:23.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509484 2024-04-25T08:41:23.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:23.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:41:23.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509484 2024-04-25T08:41:23.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:23.531 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509484 2024-04-25T08:41:23.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509484 2024-04-25T08:41:23.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509484' 2024-04-25T08:41:23.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:41:24.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509484 -lt 64424509484 2024-04-25T08:41:24.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:24.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345965 2024-04-25T08:41:24.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:24.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:41:24.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345965 2024-04-25T08:41:24.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:24.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345965 2024-04-25T08:41:24.137 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345965 2024-04-25T08:41:24.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345965' 2024-04-25T08:41:24.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:41:24.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345965 -lt 85899345965 2024-04-25T08:41:24.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:24.566 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182431 2024-04-25T08:41:24.566 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:24.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:41:24.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182431 2024-04-25T08:41:24.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:24.570 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182431 2024-04-25T08:41:24.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182431 2024-04-25T08:41:24.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182431' 2024-04-25T08:41:24.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:41:25.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182431 -lt 107374182431 2024-04-25T08:41:25.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:41:25.149 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018902 2024-04-25T08:41:25.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:41:25.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:41:25.152 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018902 2024-04-25T08:41:25.152 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:41:25.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018902 2024-04-25T08:41:25.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018902' 2024-04-25T08:41:25.154 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018902 2024-04-25T08:41:25.154 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:41:25.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018903 -lt 128849018902 2024-04-25T08:41:25.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:41:25.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:25.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:26.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:41:26.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:41:26.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:41:26.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:41:26.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:41:26.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:41:26.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:41:26.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:41:26.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:41:26.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:41:26.716 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:26.716 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:27.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:41:27.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:41:27.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:41:27.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:41:27.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:41:27.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:41:27.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:41:27.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:41:27.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:41:27.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:41:27.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:41:27.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:41:27.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:41:27.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:41:27.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:41:27.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:41:27.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:27.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:28.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:41:28.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:41:28.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:41:28.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:41:28.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:41:28.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:41:28.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:41:28.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:41:28.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:41:29.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=10 2024-04-25T08:41:29.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 10 '!=' null 2024-04-25T08:41:29.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:41:29.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:41:29.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:41:29.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:41:29.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:41:29.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:41:29.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:41:29.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:41:29.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:41:29.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:41:29.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:41:29.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:41:29.547 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:41:29.547 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:29.547 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:30.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:41:30.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:41:30.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:41:30.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:41:30.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:41:30.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:41:30.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:41:30.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:41:30.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:41:30.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:41:30.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:41:30.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:41:30.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:41:30.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:41:30.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:41:30.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:41:30.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:41:30.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:41:30.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:41:30.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:41:30.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:41:30.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:41:31.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:41:31.372 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:41:31.373 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:31.373 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:32.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:41:32.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:41:32.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:41:32.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:41:32.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:41:32.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:41:32.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:41:32.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:41:32.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:41:32.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T08:41:32.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T08:41:32.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:41:32.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:41:32.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:41:32.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:41:32.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:41:32.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:41:32.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:41:32.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:41:32.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:41:32.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:41:32.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:41:33.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:41:33.208 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:41:33.209 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:33.209 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:33.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:41:33.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:41:33.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:41:33.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:41:33.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:41:33.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:41:33.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:41:33.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:41:33.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:41:34.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-04-25T08:41:34.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-04-25T08:41:34.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:41:34.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:41:34.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:41:34.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:41:34.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:41:34.598 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:41:34.598 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:41:34.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:41:34.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:41:34.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:41:34.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:41:35.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:41:35.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:41:35.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:35.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:35.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:41:35.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:41:35.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:41:35.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:41:35.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:41:35.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:41:35.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:41:35.777 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:41:35.777 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:41:36.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:41:36.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:41:36.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:41:36.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:41:36.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:41:36.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:41:36.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:41:36.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:41:36.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:41:36.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:41:36.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:41:36.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:41:36.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:41:36.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:41:36.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:41:36.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:36.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:37.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:41:37.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:41:37.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:41:37.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:41:37.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:41:37.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:41:37.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:41:37.362 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:41:37.362 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:41:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-04-25T08:41:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-04-25T08:41:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:41:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:41:37.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:41:38.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:41:38.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:41:38.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:41:38.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:41:38.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:41:38.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:41:38.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:41:38.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:41:38.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:41:38.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:41:38.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:41:38.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:41:38.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:41:38.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:41:38.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 erasure 200 0 0 0 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-04-25T08:41:38.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-04-25T08:41:38.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:41:38.950 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:41:38.950 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:41:38.950 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:41:38.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,2]/[1,NONE,2]' 2024-04-25T08:41:38.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:41:38.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:41:38.971 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:41:38.971 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-04-25T08:41:38.971 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:41:38.971 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:41:38.971 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:41:38.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:41:38.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:41:38.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:41:38.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:41:38.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:41:38.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:41:38.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:41:38.980 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:41:38.980 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:41:38.980 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:41:38.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:41:38.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:41:38.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:41:38.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:41:38.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:41:38.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:41:38.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:41:38.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:41:38.996 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:41:38.996 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,2]/[1,NONE,2]p' 2024-04-25T08:41:38.996 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:41:38.996 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:41:38.996 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:41:39.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:41:39.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:41:39.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:41:39.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:41:39.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:41:39.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:41:39.004 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:41:39.004 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:41:39.004 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:41:39.004 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:41:39.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:41:39.564 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T08:41:39.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-04-25T08:41:39.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:41:39.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:41:39.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:41:39.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:41:39.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:41:44.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:41:44.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:41:44.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:41:44.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:41:44.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:41:44.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:41:44.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:41:44.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:41:44.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:41:44.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:41:44.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:41:44.912 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:41:44.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:41:44.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:41:44.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:41:44.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:41:44.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:41:44.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:41:44.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:41:44.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:41:44.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:41:44.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:41:44.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:41:44.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:41:44.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:41:44.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:44.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:41:44.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T08:41:44.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:41:44.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:41:44.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:41:44.966 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:41:44.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:41:44.967 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:41:44.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:41:44.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:41:44.968 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:41:44.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:41:44.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:41:44.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:41:44.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:41:44.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:41:44.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:41:44.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:41:44.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:41:44.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:44.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:41:44.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T08:41:44.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:41:44.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:41:44.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:41:44.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:41:44.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:44.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:41:44.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.280166 2024-04-25T08:41:44.977 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:41:44.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:41:44.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:41:44.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:41:44.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_erasure_remapped td/osd-recovery-stats 2024-04-25T08:41:44.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:384: TEST_recovery_erasure_remapped: local dir=td/osd-recovery-stats 2024-04-25T08:41:44.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:386: TEST_recovery_erasure_remapped: run_mon td/osd-recovery-stats a 2024-04-25T08:41:44.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:41:44.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:41:44.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:41:44.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:41:44.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:41:44.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:41:45.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:41:45.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:41:45.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:41:45.434 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:41:45.434 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:45.434 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:41:45.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:41:45.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:41:45.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:41:45.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:41:45.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:41:45.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:41:45.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:41:45.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:41:45.644 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:41:45.644 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:41:45.644 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:41:45.645 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:41:45.645 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:45.646 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:41:45.646 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:41:45.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:41:45.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get fsid 2024-04-25T08:41:45.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:41:45.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:41:45.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:41:45.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:41:45.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:41:45.761 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:41:45.762 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:41:45.762 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:41:45.762 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:41:45.762 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:45.762 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:41:45.762 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:41:45.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:41:45.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get mon_host 2024-04-25T08:41:45.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:387: TEST_recovery_erasure_remapped: run_mgr td/osd-recovery-stats x 2024-04-25T08:41:45.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:41:45.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:41:45.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:41:45.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:41:45.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:41:45.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:41:46.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:41:46.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:41:46.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:41:46.773 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:41:46.773 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:46.774 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:41:46.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:41:46.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:41:46.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:41:46.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:388: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 0 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:41:46.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:41:46.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:41:46.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:41:46.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:41:46.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:41:46.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:41:46.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7181f45f-ab15-48e4-8c64-1e132dc60b0d 2024-04-25T08:41:46.814 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 7181f45f-ab15-48e4-8c64-1e132dc60b0d 2024-04-25T08:41:46.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7181f45f-ab15-48e4-8c64-1e132dc60b0d' 2024-04-25T08:41:46.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:41:46.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBKFypmZa1rMhAAvKumo10mcozp4JsucGLC4A== 2024-04-25T08:41:46.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBKFypmZa1rMhAAvKumo10mcozp4JsucGLC4A=="}' 2024-04-25T08:41:46.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7181f45f-ab15-48e4-8c64-1e132dc60b0d -i td/osd-recovery-stats/0/new.json 2024-04-25T08:41:47.190 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:41:47.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:41:47.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBKFypmZa1rMhAAvKumo10mcozp4JsucGLC4A== --osd-uuid 7181f45f-ab15-48e4-8c64-1e132dc60b0d 2024-04-25T08:41:47.244 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:47.250+0000 7fce8d1197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:47.283 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:47.286+0000 7fce8d1197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:47.284 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:47.290+0000 7fce8d1197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:47.284 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:47.290+0000 7fce8d1197c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:41:47.284 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:47.290+0000 7fce8d1197c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:41:47.285 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:47.290+0000 7fce8d1197c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:41:47.323 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:47.326+0000 7fce8d1197c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:41:53.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:41:53.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:41:53.571 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:41:53.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:41:53.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:41:54.108 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:41:54.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:41:54.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:41:54.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:41:54.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:41:54.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:41:54.139 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:54.146+0000 7f8e97b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:54.191 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:54.198+0000 7f8e97b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:54.192 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:54.202+0000 7f8e97b1d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:41:54.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:41:54.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:41:54.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:41:54.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:41:54.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:41:54.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:54.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:41:54.546 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:41:54.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:54.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:41:54.754 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:54.762+0000 7f8e97b1d7c0 -1 Falling back to public interface 2024-04-25T08:41:54.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:55.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:55.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:55.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:41:55.999 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:41:55.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:55.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:41:56.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:57.397 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:41:57.405+0000 7f8e97b1d7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:41:57.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:57.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:57.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:41:57.405 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:41:57.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:57.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:41:58.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:41:59.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:41:59.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:41:59.088 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:41:59.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:41:59.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:41:59.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:41:59.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:00.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:00.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:00.522 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:42:00.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:42:00.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:00.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:00.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:01.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:01.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:01.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:42:01.980 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:42:01.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:01.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:02.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:03.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:03.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:03.500 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:42:03.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:42:03.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:03.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:04.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:04.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:04.974 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:42:04.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:42:04.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:04.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:05.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:06.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:06.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:06.520 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:42:06.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:42:06.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:06.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:06.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:07.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:07.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:07.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:42:07.941 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:42:07.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:07.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:08.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:09.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:09.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:09.377 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:42:09.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:42:09.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:09.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:09.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:10.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:10.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:10.800 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:42:10.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:42:10.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:10.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:11.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:13.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:13.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:13.001 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:42:13.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:42:13.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:13.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:13.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:14.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:14.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:14.830 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:42:14.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:42:14.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:14.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:15.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:16.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:16.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:16.611 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:42:16.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:42:16.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:16.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:17.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:18.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:18.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:18.579 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:42:18.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:42:18.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:18.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:19.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:20.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:20.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:20.407 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:42:20.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:42:20.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:20.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:21.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:22.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:22.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:22.330 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:42:22.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:42:22.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:22.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:22.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:23.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:23.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:23.760 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:42:23.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:42:23.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:23.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:42:24.204 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1031132236,v1:127.0.0.1:6803/1031132236] [v2:127.0.0.1:6804/1031132236,v1:127.0.0.1:6805/1031132236] exists,up 7181f45f-ab15-48e4-8c64-1e132dc60b0d 2024-04-25T08:42:24.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:42:24.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:42:24.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:42:24.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:389: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 1 2024-04-25T08:42:24.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:42:24.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:42:24.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:42:24.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:42:24.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:42:24.209 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:42:24.209 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:42:24.209 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:42:24.209 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:42:24.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:42:24.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:42:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:42:24.211 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:42:24.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=93699618-7f18-45ad-8c3c-da67860b472d 2024-04-25T08:42:24.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 93699618-7f18-45ad-8c3c-da67860b472d' 2024-04-25T08:42:24.211 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 93699618-7f18-45ad-8c3c-da67860b472d 2024-04-25T08:42:24.212 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:42:24.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwFypmqGxsDhAAR7YdGDh5aiJpB2gdeCQTcw== 2024-04-25T08:42:24.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwFypmqGxsDhAAR7YdGDh5aiJpB2gdeCQTcw=="}' 2024-04-25T08:42:24.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 93699618-7f18-45ad-8c3c-da67860b472d -i td/osd-recovery-stats/1/new.json 2024-04-25T08:42:24.755 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:42:24.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:42:24.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwFypmqGxsDhAAR7YdGDh5aiJpB2gdeCQTcw== --osd-uuid 93699618-7f18-45ad-8c3c-da67860b472d 2024-04-25T08:42:24.809 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:24.817+0000 7f61bb7c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:42:24.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:24.849+0000 7f61bb7c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:42:24.842 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:24.849+0000 7f61bb7c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:42:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:24.849+0000 7f61bb7c37c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:42:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:24.849+0000 7f61bb7c37c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:42:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:24.849+0000 7f61bb7c37c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:42:24.880 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:24.889+0000 7f61bb7c37c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:42:31.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:42:31.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:42:31.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:42:31.558 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:42:31.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:42:32.110 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:42:32.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:42:32.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:42:32.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:42:32.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:42:32.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:42:32.142 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:32.149+0000 7f6a0ac347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:42:32.156 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:32.165+0000 7f6a0ac347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:42:32.158 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:32.165+0000 7f6a0ac347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:42:32.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:42:32.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:42:32.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:42:32.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:42:32.561 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:42:32.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:42:32.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:32.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:42:32.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:32.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:32.709 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:32.717+0000 7f6a0ac347c0 -1 Falling back to public interface 2024-04-25T08:42:32.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:33.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:33.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:33.988 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:42:33.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:42:33.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:33.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:34.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:35.305 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:42:35.313+0000 7f6a0ac347c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:42:35.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:35.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:35.400 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:42:35.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:42:35.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:35.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:36.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:37.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:37.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:37.027 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:42:37.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:42:37.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:37.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:37.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:38.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:38.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:38.557 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:42:38.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:42:38.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:38.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:39.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:40.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:40.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:40.029 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:42:40.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:42:40.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:40.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:40.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:41.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:41.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:41.602 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:42:41.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:42:41.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:41.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:42.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:43.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:43.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:43.257 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:42:43.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:42:43.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:43.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:43.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:44.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:44.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:44.714 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:42:44.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:42:44.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:44.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:45.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:46.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:46.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:46.190 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:42:46.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:42:46.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:46.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:46.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:47.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:47.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:47.612 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:42:47.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:42:47.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:47.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:48.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:49.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:49.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:49.170 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:42:49.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:42:49.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:49.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:50.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:51.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:51.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:51.203 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:42:51.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:42:51.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:51.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:52.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:53.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:53.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:53.361 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:42:53.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:42:53.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:53.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:54.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:55.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:55.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:55.087 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:42:55.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:42:55.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:55.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:55.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:57.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:42:57.002 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:42:57.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:57.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:57.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:42:58.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:42:58.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:42:58.692 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:42:58.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:42:58.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:42:58.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:42:59.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:00.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:00.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:00.533 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:43:00.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:43:00.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:00.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:43:01.039 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/795815633,v1:127.0.0.1:6811/795815633] [v2:127.0.0.1:6812/795815633,v1:127.0.0.1:6813/795815633] exists,up 93699618-7f18-45ad-8c3c-da67860b472d 2024-04-25T08:43:01.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:43:01.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:43:01.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:43:01.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:390: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 2 2024-04-25T08:43:01.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:43:01.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:43:01.042 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:43:01.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:01.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:01.043 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:01.043 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:01.043 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:43:01.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:43:01.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:43:01.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:43:01.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:43:01.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:43:01.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:43:01.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8417bd5a-d83d-409f-9a75-b13611e9d20d 2024-04-25T08:43:01.045 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 8417bd5a-d83d-409f-9a75-b13611e9d20d 2024-04-25T08:43:01.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 8417bd5a-d83d-409f-9a75-b13611e9d20d' 2024-04-25T08:43:01.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:43:01.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCVFypmO114BBAAAS1bittfZbHepl/L2qyuhQ== 2024-04-25T08:43:01.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCVFypmO114BBAAAS1bittfZbHepl/L2qyuhQ=="}' 2024-04-25T08:43:01.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8417bd5a-d83d-409f-9a75-b13611e9d20d -i td/osd-recovery-stats/2/new.json 2024-04-25T08:43:01.837 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:43:01.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:43:01.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCVFypmO114BBAAAS1bittfZbHepl/L2qyuhQ== --osd-uuid 8417bd5a-d83d-409f-9a75-b13611e9d20d 2024-04-25T08:43:01.898 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:01.904+0000 7f38cfa627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:01.910 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:01.916+0000 7f38cfa627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:01.911 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:01.920+0000 7f38cfa627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:01.912 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:01.920+0000 7f38cfa627c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:43:01.912 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:01.920+0000 7f38cfa627c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:43:01.912 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:01.920+0000 7f38cfa627c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:43:01.985 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:01.992+0000 7f38cfa627c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:43:08.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:43:08.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:43:08.829 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T08:43:08.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:43:08.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:43:09.409 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T08:43:09.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:43:09.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:43:09.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:43:09.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:43:09.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:43:09.441 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:09.448+0000 7faa62e8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:09.458 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:09.464+0000 7faa62e8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:09.459 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:09.468+0000 7faa62e8b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:09.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:43:09.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:43:09.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:43:09.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:43:09.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:43:09.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:09.847 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:43:09.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:43:09.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:09.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:10.037 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:10.044+0000 7faa62e8b7c0 -1 Falling back to public interface 2024-04-25T08:43:10.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:11.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:11.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:11.281 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:43:11.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:43:11.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:11.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:11.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:12.656 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:12.664+0000 7faa62e8b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:43:12.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:12.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:12.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:43:12.696 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:43:12.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:12.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:13.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:14.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:14.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:14.570 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:43:14.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:43:14.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:14.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:15.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:16.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:16.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:16.024 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:43:16.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:43:16.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:16.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:16.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:17.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:17.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:17.598 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:43:17.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:43:17.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:17.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:18.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:19.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:19.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:19.176 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:43:19.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:43:19.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:19.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:19.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:20.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:20.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:20.813 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:43:20.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:43:20.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:20.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:21.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:22.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:22.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:22.244 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:43:22.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:43:22.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:22.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:22.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:23.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:23.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:23.744 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:43:23.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:43:23.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:23.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:24.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:25.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:25.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:25.252 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:43:25.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:43:25.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:25.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:25.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:26.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:26.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:26.870 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:43:26.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:43:26.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:26.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:27.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:28.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:28.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:28.525 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:43:28.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:43:28.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:28.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:29.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:30.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:30.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:30.080 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:43:30.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:43:30.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:30.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:31.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:32.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:32.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:32.035 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:43:32.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:43:32.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:32.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:32.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:33.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:33.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:33.738 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:43:33.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:43:33.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:33.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:34.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:35.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:35.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:35.259 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:43:35.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:43:35.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:35.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:36.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:37.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:37.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:37.368 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:43:37.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:43:37.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:37.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:38.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:39.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:39.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:39.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:43:39.007 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:43:39.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:39.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:43:39.422 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3496867809,v1:127.0.0.1:6819/3496867809] [v2:127.0.0.1:6820/3496867809,v1:127.0.0.1:6821/3496867809] exists,up 8417bd5a-d83d-409f-9a75-b13611e9d20d 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:391: TEST_recovery_erasure_remapped: run_osd td/osd-recovery-stats 3 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:43:39.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:43:39.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:43:39.428 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:43:39.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=708a9b8a-791f-4c4c-96b9-be5afa46eee3 2024-04-25T08:43:39.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 708a9b8a-791f-4c4c-96b9-be5afa46eee3' 2024-04-25T08:43:39.429 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 708a9b8a-791f-4c4c-96b9-be5afa46eee3 2024-04-25T08:43:39.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:43:39.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC7FypmTxsBGxAAdKlWk5rcltLODl7d/+Mugw== 2024-04-25T08:43:39.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC7FypmTxsBGxAAdKlWk5rcltLODl7d/+Mugw=="}' 2024-04-25T08:43:39.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 708a9b8a-791f-4c4c-96b9-be5afa46eee3 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:43:39.951 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:43:39.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:43:39.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC7FypmTxsBGxAAdKlWk5rcltLODl7d/+Mugw== --osd-uuid 708a9b8a-791f-4c4c-96b9-be5afa46eee3 2024-04-25T08:43:39.995 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:39.999+0000 7f7a007aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:40.012 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:40.019+0000 7f7a007aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:40.014 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:40.019+0000 7f7a007aa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:40.014 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:40.019+0000 7f7a007aa7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:43:40.014 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:40.019+0000 7f7a007aa7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:43:40.015 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:40.019+0000 7f7a007aa7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:43:40.054 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:40.059+0000 7f7a007aa7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:43:46.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:43:46.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:43:46.897 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T08:43:46.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:43:46.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:43:47.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:43:47.475 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T08:43:47.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:43:47.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:43:47.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:43:47.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:43:47.508 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:47.515+0000 7f07a47517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:47.516 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:47.523+0000 7f07a47517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:47.517 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:47.523+0000 7f07a47517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:43:47.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:43:47.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:43:47.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:43:47.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:43:47.871 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:43:47.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:43:47.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:47.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:43:47.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:47.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:43:48.082 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:48.087+0000 7f07a47517c0 -1 Falling back to public interface 2024-04-25T08:43:48.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:49.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:49.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:49.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:43:49.279 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:43:49.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:49.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:43:49.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:50.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:50.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:50.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:43:50.685 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:43:50.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:50.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:43:50.705 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:43:50.711+0000 7f07a47517c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:43:51.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:52.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:52.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:52.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:43:52.256 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:43:52.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:52.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:43:52.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:53.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:53.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:53.781 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:43:53.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:43:53.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:53.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:43:54.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:55.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:55.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:55.357 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:43:55.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:43:55.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:55.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:43:55.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:56.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:56.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:56.973 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:43:56.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:43:56.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:56.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:43:57.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:58.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:58.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:58.401 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:43:58.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:43:58.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:58.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:43:58.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:43:59.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:43:59.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:43:59.893 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:43:59.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:43:59.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:43:59.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:00.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:01.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:01.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:01.365 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:44:01.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:44:01.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:01.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:02.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:03.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:03.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:03.210 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:44:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:44:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:03.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:03.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:04.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:04.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:04.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:44:04.881 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:44:04.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:04.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:05.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:06.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:06.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:06.569 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:44:06.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:44:06.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:06.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:07.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:08.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:08.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:08.205 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:44:08.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:44:08.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:08.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:09.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:10.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:10.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:10.350 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:44:10.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:44:10.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:10.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:11.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:12.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:12.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:12.187 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:44:12.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:44:12.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:12.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:13.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:14.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:14.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:14.117 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:44:14.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:44:14.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:14.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:15.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:16.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:16.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:16.395 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:44:16.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:44:16.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:16.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:16.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:44:17.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:44:17.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:44:17.871 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:44:17.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:44:17.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:44:17.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:44:18.306 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/943002133,v1:127.0.0.1:6827/943002133] [v2:127.0.0.1:6828/943002133,v1:127.0.0.1:6829/943002133] exists,up 708a9b8a-791f-4c4c-96b9-be5afa46eee3 2024-04-25T08:44:18.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:44:18.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:44:18.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:44:18.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:393: TEST_recovery_erasure_remapped: ceph osd erasure-code-profile set myprofile plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-04-25T08:44:18.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:394: TEST_recovery_erasure_remapped: create_pool test 1 1 erasure myprofile 2024-04-25T08:44:18.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 erasure myprofile 2024-04-25T08:44:19.921 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T08:44:19.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:44:20.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:395: TEST_recovery_erasure_remapped: ceph osd pool set test min_size 2 2024-04-25T08:44:21.939 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 min_size to 2 2024-04-25T08:44:21.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:397: TEST_recovery_erasure_remapped: wait_for_clean 2024-04-25T08:44:21.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:44:21.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:44:21.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:44:21.964 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:44:21.964 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:44:21.964 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:44:21.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:44:21.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:44:21.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:44:22.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:44:22.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:44:22.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:44:22.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:44:22.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:44:22.081 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:44:22.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:44:22.588 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:44:22.588 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:44:22.588 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T08:44:22.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:44:22.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:44:22.588 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:44:22.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836511 2024-04-25T08:44:22.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836511 2024-04-25T08:44:22.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511' 2024-04-25T08:44:22.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:44:22.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:44:22.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672984 2024-04-25T08:44:22.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672984 2024-04-25T08:44:22.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672984' 2024-04-25T08:44:22.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:44:22.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:44:23.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-04-25T08:44:23.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-04-25T08:44:23.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672984 2-64424509458' 2024-04-25T08:44:23.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:44:23.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:44:23.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345929 2024-04-25T08:44:23.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345929 2024-04-25T08:44:23.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836511 1-42949672984 2-64424509458 3-85899345929' 2024-04-25T08:44:23.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:44:23.195 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836511 2024-04-25T08:44:23.195 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:44:23.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:44:23.196 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836511 2024-04-25T08:44:23.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:44:23.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836511 2024-04-25T08:44:23.197 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836511 2024-04-25T08:44:23.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-04-25T08:44:23.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:44:23.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836511 -lt 21474836511 2024-04-25T08:44:23.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:44:23.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672984 2024-04-25T08:44:23.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:44:23.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:44:23.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:44:23.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672984 2024-04-25T08:44:23.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672984 2024-04-25T08:44:23.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2024-04-25T08:44:23.626 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672984 2024-04-25T08:44:23.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:44:24.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672984 -lt 42949672984 2024-04-25T08:44:24.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:44:24.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-04-25T08:44:24.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:44:24.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:44:24.024 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-04-25T08:44:24.024 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:44:24.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-04-25T08:44:24.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-04-25T08:44:24.026 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509458 2024-04-25T08:44:24.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:44:24.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-04-25T08:44:24.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:44:24.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345929 2024-04-25T08:44:24.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:44:24.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:44:24.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345929 2024-04-25T08:44:24.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:44:24.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345929 2024-04-25T08:44:24.502 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345929 2024-04-25T08:44:24.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345929' 2024-04-25T08:44:24.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:44:24.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345929 -lt 85899345929 2024-04-25T08:44:24.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:44:24.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:44:24.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:44:25.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:44:25.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:44:25.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:44:25.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:44:25.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:44:25.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:44:25.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:44:25.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:44:25.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:44:25.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:44:25.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:44:25.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:44:26.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:44:26.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:44:26.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:44:26.459 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: seq 1 200 2024-04-25T08:44:26.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:26.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj1 /dev/null 2024-04-25T08:44:26.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:26.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj2 /dev/null 2024-04-25T08:44:26.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:26.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj3 /dev/null 2024-04-25T08:44:27.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:27.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj4 /dev/null 2024-04-25T08:44:27.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:27.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj5 /dev/null 2024-04-25T08:44:27.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:27.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj6 /dev/null 2024-04-25T08:44:27.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:27.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj7 /dev/null 2024-04-25T08:44:27.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:27.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj8 /dev/null 2024-04-25T08:44:28.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:28.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj9 /dev/null 2024-04-25T08:44:28.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:28.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj10 /dev/null 2024-04-25T08:44:28.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:28.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj11 /dev/null 2024-04-25T08:44:28.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:28.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj12 /dev/null 2024-04-25T08:44:28.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:28.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj13 /dev/null 2024-04-25T08:44:29.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:29.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj14 /dev/null 2024-04-25T08:44:29.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:29.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj15 /dev/null 2024-04-25T08:44:29.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:29.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj16 /dev/null 2024-04-25T08:44:29.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:29.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj17 /dev/null 2024-04-25T08:44:30.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:30.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj18 /dev/null 2024-04-25T08:44:30.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:30.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj19 /dev/null 2024-04-25T08:44:30.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:30.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj20 /dev/null 2024-04-25T08:44:30.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:30.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj21 /dev/null 2024-04-25T08:44:30.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:30.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj22 /dev/null 2024-04-25T08:44:31.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:31.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj23 /dev/null 2024-04-25T08:44:31.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:31.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj24 /dev/null 2024-04-25T08:44:31.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:31.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj25 /dev/null 2024-04-25T08:44:31.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:31.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj26 /dev/null 2024-04-25T08:44:31.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:31.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj27 /dev/null 2024-04-25T08:44:32.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:32.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj28 /dev/null 2024-04-25T08:44:32.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:32.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj29 /dev/null 2024-04-25T08:44:32.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:32.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj30 /dev/null 2024-04-25T08:44:32.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:32.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj31 /dev/null 2024-04-25T08:44:32.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:32.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj32 /dev/null 2024-04-25T08:44:33.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:33.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj33 /dev/null 2024-04-25T08:44:33.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:33.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj34 /dev/null 2024-04-25T08:44:33.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:33.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj35 /dev/null 2024-04-25T08:44:33.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:33.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj36 /dev/null 2024-04-25T08:44:33.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:33.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj37 /dev/null 2024-04-25T08:44:34.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:34.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj38 /dev/null 2024-04-25T08:44:34.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:34.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj39 /dev/null 2024-04-25T08:44:34.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:34.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj40 /dev/null 2024-04-25T08:44:34.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:34.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj41 /dev/null 2024-04-25T08:44:34.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:34.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj42 /dev/null 2024-04-25T08:44:35.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:35.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj43 /dev/null 2024-04-25T08:44:35.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:35.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj44 /dev/null 2024-04-25T08:44:35.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:35.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj45 /dev/null 2024-04-25T08:44:35.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:35.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj46 /dev/null 2024-04-25T08:44:35.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:35.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj47 /dev/null 2024-04-25T08:44:36.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:36.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj48 /dev/null 2024-04-25T08:44:36.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:36.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj49 /dev/null 2024-04-25T08:44:36.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:36.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj50 /dev/null 2024-04-25T08:44:36.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:36.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj51 /dev/null 2024-04-25T08:44:37.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:37.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj52 /dev/null 2024-04-25T08:44:37.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:37.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj53 /dev/null 2024-04-25T08:44:37.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:37.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj54 /dev/null 2024-04-25T08:44:37.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:37.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj55 /dev/null 2024-04-25T08:44:37.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:37.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj56 /dev/null 2024-04-25T08:44:38.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:38.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj57 /dev/null 2024-04-25T08:44:38.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:38.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj58 /dev/null 2024-04-25T08:44:38.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:38.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj59 /dev/null 2024-04-25T08:44:38.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:38.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj60 /dev/null 2024-04-25T08:44:38.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:38.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj61 /dev/null 2024-04-25T08:44:39.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:39.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj62 /dev/null 2024-04-25T08:44:39.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:39.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj63 /dev/null 2024-04-25T08:44:39.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:39.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj64 /dev/null 2024-04-25T08:44:39.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:39.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj65 /dev/null 2024-04-25T08:44:39.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:39.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj66 /dev/null 2024-04-25T08:44:40.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:40.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj67 /dev/null 2024-04-25T08:44:40.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:40.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj68 /dev/null 2024-04-25T08:44:40.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:40.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj69 /dev/null 2024-04-25T08:44:40.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:40.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj70 /dev/null 2024-04-25T08:44:41.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:41.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj71 /dev/null 2024-04-25T08:44:41.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:41.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj72 /dev/null 2024-04-25T08:44:41.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:41.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj73 /dev/null 2024-04-25T08:44:41.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:41.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj74 /dev/null 2024-04-25T08:44:41.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:41.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj75 /dev/null 2024-04-25T08:44:42.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:42.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj76 /dev/null 2024-04-25T08:44:42.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:42.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj77 /dev/null 2024-04-25T08:44:42.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:42.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj78 /dev/null 2024-04-25T08:44:42.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:42.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj79 /dev/null 2024-04-25T08:44:42.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:42.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj80 /dev/null 2024-04-25T08:44:43.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:43.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj81 /dev/null 2024-04-25T08:44:43.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:43.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj82 /dev/null 2024-04-25T08:44:43.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:43.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj83 /dev/null 2024-04-25T08:44:43.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:43.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj84 /dev/null 2024-04-25T08:44:43.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:43.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj85 /dev/null 2024-04-25T08:44:44.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:44.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj86 /dev/null 2024-04-25T08:44:44.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:44.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj87 /dev/null 2024-04-25T08:44:44.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:44.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj88 /dev/null 2024-04-25T08:44:44.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:44.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj89 /dev/null 2024-04-25T08:44:45.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:45.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj90 /dev/null 2024-04-25T08:44:45.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:45.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj91 /dev/null 2024-04-25T08:44:45.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:45.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj92 /dev/null 2024-04-25T08:44:45.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:45.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj93 /dev/null 2024-04-25T08:44:45.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:45.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj94 /dev/null 2024-04-25T08:44:46.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:46.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj95 /dev/null 2024-04-25T08:44:46.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:46.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj96 /dev/null 2024-04-25T08:44:46.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:46.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj97 /dev/null 2024-04-25T08:44:46.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:46.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj98 /dev/null 2024-04-25T08:44:46.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:46.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj99 /dev/null 2024-04-25T08:44:47.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:47.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj100 /dev/null 2024-04-25T08:44:47.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:47.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj101 /dev/null 2024-04-25T08:44:47.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:47.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj102 /dev/null 2024-04-25T08:44:47.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:47.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj103 /dev/null 2024-04-25T08:44:47.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:47.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj104 /dev/null 2024-04-25T08:44:48.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:48.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj105 /dev/null 2024-04-25T08:44:48.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:48.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj106 /dev/null 2024-04-25T08:44:48.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:48.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj107 /dev/null 2024-04-25T08:44:48.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:48.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj108 /dev/null 2024-04-25T08:44:48.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:48.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj109 /dev/null 2024-04-25T08:44:49.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:49.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj110 /dev/null 2024-04-25T08:44:49.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:49.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj111 /dev/null 2024-04-25T08:44:49.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:49.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj112 /dev/null 2024-04-25T08:44:49.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:49.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj113 /dev/null 2024-04-25T08:44:50.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:50.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj114 /dev/null 2024-04-25T08:44:50.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:50.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj115 /dev/null 2024-04-25T08:44:50.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:50.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj116 /dev/null 2024-04-25T08:44:51.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:51.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj117 /dev/null 2024-04-25T08:44:51.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:51.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj118 /dev/null 2024-04-25T08:44:51.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:51.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj119 /dev/null 2024-04-25T08:44:51.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:51.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj120 /dev/null 2024-04-25T08:44:51.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:51.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj121 /dev/null 2024-04-25T08:44:52.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:52.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj122 /dev/null 2024-04-25T08:44:52.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:52.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj123 /dev/null 2024-04-25T08:44:52.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:52.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj124 /dev/null 2024-04-25T08:44:52.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:52.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj125 /dev/null 2024-04-25T08:44:53.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:53.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj126 /dev/null 2024-04-25T08:44:53.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:53.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj127 /dev/null 2024-04-25T08:44:53.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:53.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj128 /dev/null 2024-04-25T08:44:53.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:53.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj129 /dev/null 2024-04-25T08:44:53.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:53.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj130 /dev/null 2024-04-25T08:44:54.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:54.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj131 /dev/null 2024-04-25T08:44:54.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:54.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj132 /dev/null 2024-04-25T08:44:54.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:54.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj133 /dev/null 2024-04-25T08:44:54.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:54.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj134 /dev/null 2024-04-25T08:44:54.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:54.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj135 /dev/null 2024-04-25T08:44:55.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:55.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj136 /dev/null 2024-04-25T08:44:55.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:55.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj137 /dev/null 2024-04-25T08:44:55.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:55.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj138 /dev/null 2024-04-25T08:44:55.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:55.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj139 /dev/null 2024-04-25T08:44:56.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:56.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj140 /dev/null 2024-04-25T08:44:56.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:56.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj141 /dev/null 2024-04-25T08:44:56.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:56.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj142 /dev/null 2024-04-25T08:44:56.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:56.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj143 /dev/null 2024-04-25T08:44:56.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:56.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj144 /dev/null 2024-04-25T08:44:57.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:57.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj145 /dev/null 2024-04-25T08:44:57.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:57.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj146 /dev/null 2024-04-25T08:44:57.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:57.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj147 /dev/null 2024-04-25T08:44:57.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:57.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj148 /dev/null 2024-04-25T08:44:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj149 /dev/null 2024-04-25T08:44:58.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:58.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj150 /dev/null 2024-04-25T08:44:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj151 /dev/null 2024-04-25T08:44:58.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:58.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj152 /dev/null 2024-04-25T08:44:58.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:58.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj153 /dev/null 2024-04-25T08:44:59.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:59.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj154 /dev/null 2024-04-25T08:44:59.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:59.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj155 /dev/null 2024-04-25T08:44:59.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:59.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj156 /dev/null 2024-04-25T08:44:59.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:59.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj157 /dev/null 2024-04-25T08:44:59.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:44:59.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj158 /dev/null 2024-04-25T08:45:00.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:00.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj159 /dev/null 2024-04-25T08:45:00.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:00.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj160 /dev/null 2024-04-25T08:45:00.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:00.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj161 /dev/null 2024-04-25T08:45:00.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:00.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj162 /dev/null 2024-04-25T08:45:00.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:00.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj163 /dev/null 2024-04-25T08:45:01.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:01.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj164 /dev/null 2024-04-25T08:45:01.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:01.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj165 /dev/null 2024-04-25T08:45:01.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:01.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj166 /dev/null 2024-04-25T08:45:01.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:01.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj167 /dev/null 2024-04-25T08:45:02.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:02.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj168 /dev/null 2024-04-25T08:45:02.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:02.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj169 /dev/null 2024-04-25T08:45:02.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:02.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj170 /dev/null 2024-04-25T08:45:02.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:02.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj171 /dev/null 2024-04-25T08:45:02.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:02.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj172 /dev/null 2024-04-25T08:45:03.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:03.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj173 /dev/null 2024-04-25T08:45:03.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:03.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj174 /dev/null 2024-04-25T08:45:03.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:03.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj175 /dev/null 2024-04-25T08:45:03.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:03.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj176 /dev/null 2024-04-25T08:45:03.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:03.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj177 /dev/null 2024-04-25T08:45:04.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:04.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj178 /dev/null 2024-04-25T08:45:04.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:04.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj179 /dev/null 2024-04-25T08:45:04.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:04.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj180 /dev/null 2024-04-25T08:45:04.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:04.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj181 /dev/null 2024-04-25T08:45:04.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:04.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj182 /dev/null 2024-04-25T08:45:05.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:05.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj183 /dev/null 2024-04-25T08:45:05.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:05.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj184 /dev/null 2024-04-25T08:45:05.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:05.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj185 /dev/null 2024-04-25T08:45:05.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:05.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj186 /dev/null 2024-04-25T08:45:06.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:06.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj187 /dev/null 2024-04-25T08:45:06.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:06.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj188 /dev/null 2024-04-25T08:45:06.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:06.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj189 /dev/null 2024-04-25T08:45:06.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:06.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj190 /dev/null 2024-04-25T08:45:06.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:06.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj191 /dev/null 2024-04-25T08:45:07.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:07.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj192 /dev/null 2024-04-25T08:45:07.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:07.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj193 /dev/null 2024-04-25T08:45:07.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:07.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj194 /dev/null 2024-04-25T08:45:07.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:07.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj195 /dev/null 2024-04-25T08:45:08.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:08.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj196 /dev/null 2024-04-25T08:45:08.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:08.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj197 /dev/null 2024-04-25T08:45:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj198 /dev/null 2024-04-25T08:45:08.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:08.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj199 /dev/null 2024-04-25T08:45:08.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:399: TEST_recovery_erasure_remapped: for i in $(seq 1 $objects) 2024-04-25T08:45:08.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:401: TEST_recovery_erasure_remapped: rados -p test put obj200 /dev/null 2024-04-25T08:45:09.123 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: get_primary test obj1 2024-04-25T08:45:09.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:45:09.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:45:09.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:45:09.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:45:09.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:404: TEST_recovery_erasure_remapped: local primary=1 2024-04-25T08:45:09.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: get_pg test obj1 2024-04-25T08:45:09.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:45:09.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:45:09.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:45:09.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:45:10.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:405: TEST_recovery_erasure_remapped: local PG=1.0 2024-04-25T08:45:10.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: get_not_primary test obj1 2024-04-25T08:45:10.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:45:10.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:45:10.085 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:45:10.085 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:45:10.085 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:45:10.086 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:45:10.086 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:45:10.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:45:10.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:45:10.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:45:11.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:406: TEST_recovery_erasure_remapped: local otherosd=0 2024-04-25T08:45:11.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:408: TEST_recovery_erasure_remapped: ceph osd set norecover 2024-04-25T08:45:11.820 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is set 2024-04-25T08:45:11.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:45:11.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:409: TEST_recovery_erasure_remapped: kill 297890 2024-04-25T08:45:11.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:410: TEST_recovery_erasure_remapped: ceph osd down osd.0 2024-04-25T08:45:12.376 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T08:45:12.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:411: TEST_recovery_erasure_remapped: ceph osd out osd.0 2024-04-25T08:45:13.043 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 0 = 1 ']' 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 0 = 0 ']' 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:422: TEST_recovery_erasure_remapped: continue 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 1 = 1 ']' 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:418: TEST_recovery_erasure_remapped: continue 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:414: TEST_recovery_erasure_remapped: for i in 0 1 2 3 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:416: TEST_recovery_erasure_remapped: '[' 2 = 1 ']' 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:420: TEST_recovery_erasure_remapped: '[' 2 = 0 ']' 2024-04-25T08:45:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:424: TEST_recovery_erasure_remapped: ceph osd out osd.2 2024-04-25T08:45:13.627 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already out. 2024-04-25T08:45:13.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:425: TEST_recovery_erasure_remapped: break 2024-04-25T08:45:13.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:427: TEST_recovery_erasure_remapped: ceph osd unset norecover 2024-04-25T08:45:14.625 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is unset 2024-04-25T08:45:14.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: get_primary test obj1 2024-04-25T08:45:14.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:45:14.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:45:14.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:45:14.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:45:15.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:428: TEST_recovery_erasure_remapped: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:45:15.324 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:45:15.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:429: TEST_recovery_erasure_remapped: sleep 2 2024-04-25T08:45:17.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:431: TEST_recovery_erasure_remapped: wait_for_clean 2024-04-25T08:45:17.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:45:17.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:45:17.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:45:17.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:45:17.341 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:45:17.341 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:45:17.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:45:17.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:45:17.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:45:17.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:45:17.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:45:17.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:45:17.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:45:17.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:45:17.466 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:45:17.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:45:17.974 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:45:17.974 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:45:17.974 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T08:45:17.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:45:17.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:45:17.974 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:45:18.138 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:45:18.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:45:18.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:45:18.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:45:18.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:45:18.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:45:18.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672997 2024-04-25T08:45:18.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672997 2024-04-25T08:45:18.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672997' 2024-04-25T08:45:18.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:45:18.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:45:18.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509471 2024-04-25T08:45:18.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509471 2024-04-25T08:45:18.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672997 2-64424509471' 2024-04-25T08:45:18.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:45:18.463 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:45:18.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345942 2024-04-25T08:45:18.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345942 2024-04-25T08:45:18.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949672997 2-64424509471 3-85899345942' 2024-04-25T08:45:18.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:45:18.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672997 2024-04-25T08:45:18.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:45:18.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:45:18.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672997 2024-04-25T08:45:18.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:45:18.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672997 2024-04-25T08:45:18.625 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672997 2024-04-25T08:45:18.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672997' 2024-04-25T08:45:18.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:45:19.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672995 -lt 42949672997 2024-04-25T08:45:19.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:45:20.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:45:20.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:45:20.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672997 -lt 42949672997 2024-04-25T08:45:20.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:45:20.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509471 2024-04-25T08:45:20.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:45:20.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:45:20.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509471 2024-04-25T08:45:20.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:45:20.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509471 2024-04-25T08:45:20.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-04-25T08:45:20.458 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509471 2024-04-25T08:45:20.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:45:20.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509472 -lt 64424509471 2024-04-25T08:45:20.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:45:20.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:45:20.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345942 2024-04-25T08:45:20.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:45:20.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:45:20.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345942 2024-04-25T08:45:20.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345942 2024-04-25T08:45:20.931 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345942 2024-04-25T08:45:20.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345942' 2024-04-25T08:45:20.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:45:21.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345942 -lt 85899345942 2024-04-25T08:45:21.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:45:21.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:21.953 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:22.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:45:22.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:45:22.666 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:45:22.666 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:45:22.666 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:45:22.666 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:45:22.666 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:45:22.666 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:45:23.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:45:23.145 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:45:23.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:23.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:23.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:45:23.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:45:23.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:45:23.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:45:23.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:45:23.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:45:23.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:45:23.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:45:23.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:45:23.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:45:23.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:45:23.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:45:23.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:45:23.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:45:24.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:45:24.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:45:24.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:24.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:24.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:45:24.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:45:24.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:45:24.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:45:24.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:45:24.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:45:24.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:45:24.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:45:24.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:45:25.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:45:25.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:45:25.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:45:25.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:45:25.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:45:25.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:45:25.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:45:25.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:45:25.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:45:25.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:45:25.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:45:25.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:45:25.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:45:26.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:45:26.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:45:26.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:26.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:26.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:45:26.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:45:26.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:45:26.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:45:26.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:45:26.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:45:26.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:45:26.711 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:45:26.711 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:45:27.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:45:27.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:45:27.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:45:27.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:45:27.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:45:27.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:45:27.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:45:27.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:45:27.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:45:27.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:45:27.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:45:27.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:45:27.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:45:27.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:45:27.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:45:27.933 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:27.933 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:45:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:45:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:45:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:45:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:45:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:45:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:45:28.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:45:28.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:45:29.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12 2024-04-25T08:45:29.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12 '!=' null 2024-04-25T08:45:29.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:45:29.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:45:29.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:45:29.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:45:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:45:29.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:45:29.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:45:29.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:45:29.305 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:45:29.305 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:45:29.305 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:45:29.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:45:29.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:45:29.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:29.698 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:30.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:45:30.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:45:30.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:45:30.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:45:30.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:45:30.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:45:30.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:45:30.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:45:30.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:45:30.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:45:30.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:45:30.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:45:30.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:45:30.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:45:31.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:45:31.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:45:31.071 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:45:31.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:45:31.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:45:31.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:45:31.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:45:31.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:45:31.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:45:31.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:45:31.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:31.572 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:32.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:45:32.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:45:32.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:45:32.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:45:32.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:45:32.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:45:32.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:45:32.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:45:32.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:45:32.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T08:45:32.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T08:45:32.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:45:32.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:45:32.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:45:32.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:45:32.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:45:32.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:45:32.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:45:32.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:45:32.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:45:32.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:45:32.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:45:33.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:45:33.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:45:33.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:33.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:33.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:45:33.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:45:33.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:45:33.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:45:33.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:45:33.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:45:33.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:45:33.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:45:33.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:45:34.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=14 2024-04-25T08:45:34.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 14 '!=' null 2024-04-25T08:45:34.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:45:34.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:45:34.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:45:34.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:45:34.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:45:34.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:45:34.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:45:34.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:45:34.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:45:34.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:45:34.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:45:34.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:45:34.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:45:34.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:45:34.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:433: TEST_recovery_erasure_remapped: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:434: TEST_recovery_erasure_remapped: check td/osd-recovery-stats 1.0 1 erasure 200 0 200 200 2024-04-25T08:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:45:35.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=erasure 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=200 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=200 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' erasure = erasure ']' 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:80: check: addp=p 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:45:35.380 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:45:35.381 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:45:35.381 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:45:35.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,3,NONE]/[1,NONE,2]' 2024-04-25T08:45:35.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:45:35.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:45:35.400 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:45:35.400 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-04-25T08:45:35.400 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:45:35.400 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:45:35.400 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:45:35.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:45:35.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:45:35.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:45:35.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:45:35.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:45:35.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:45:35.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:45:35.410 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:45:35.410 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:45:35.410 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:45:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:45:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:45:35.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:45:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:45:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:45:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:45:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:45:35.421 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:45:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:45:35.422 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,3,NONE]/[1,NONE,2]p' 2024-04-25T08:45:35.422 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:45:35.422 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:45:35.422 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:45:35.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=200 2024-04-25T08:45:35.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 200 200 2024-04-25T08:45:35.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:45:35.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:45:35.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:45:35.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:45:35.428 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:45:35.428 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:45:35.429 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:45:35.429 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=200 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 200 200 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=200 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=200 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:436: TEST_recovery_erasure_remapped: delete_pool test 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:45:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:45:35.949 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T08:45:35.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:437: TEST_recovery_erasure_remapped: kill_daemons td/osd-recovery-stats 2024-04-25T08:45:35.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:45:35.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:45:35.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:45:35.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:45:35.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:45:41.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:45:41.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:45:41.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:45:41.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:45:41.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:45:41.284 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:45:41.284 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:45:41.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:45:41.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:45:41.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:45:41.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:45:41.294 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:45:41.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:45:41.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:45:41.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:45:41.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:45:41.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:45:41.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:45:41.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:45:41.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:45:41.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:45:41.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:45:41.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:45:41.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:45:41.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:45:41.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:41.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:45:41.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:45:41.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:45:41.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:45:41.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:45:41.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:45:41.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:45:41.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:45:41.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:45:41.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:45:41.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:45:41.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:45:41.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:45:41.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:45:41.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:45:41.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:45:41.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:45:41.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:45:41.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:45:41.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:45:41.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:41.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:45:41.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T08:45:41.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:45:41.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:45:41.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:45:41.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:45:41.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:41.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:45:41.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.280166 2024-04-25T08:45:41.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:45:41.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:45:41.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:45:41.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:45:41.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_multi td/osd-recovery-stats 2024-04-25T08:45:41.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:441: TEST_recovery_multi: local dir=td/osd-recovery-stats 2024-04-25T08:45:41.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:443: TEST_recovery_multi: local osds=6 2024-04-25T08:45:41.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:444: TEST_recovery_multi: run_mon td/osd-recovery-stats a 2024-04-25T08:45:41.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:45:41.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:45:41.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:45:41.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:45:41.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:45:41.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:45:41.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:45:41.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:45:41.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:45:41.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:45:41.851 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:41.851 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:45:41.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:45:41.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:45:42.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:45:42.080 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:45:42.080 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:45:42.080 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:45:42.081 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:45:42.081 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:45:42.081 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:45:42.082 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:45:42.082 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:45:42.083 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:45:42.083 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:42.083 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:45:42.083 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:45:42.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:45:42.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get fsid 2024-04-25T08:45:42.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:45:42.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:45:42.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:45:42.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:45:42.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:45:42.200 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:45:42.200 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:45:42.201 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:45:42.201 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:45:42.201 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:42.201 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:45:42.202 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:45:42.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:45:42.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get mon_host 2024-04-25T08:45:42.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:445: TEST_recovery_multi: run_mgr td/osd-recovery-stats x 2024-04-25T08:45:42.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:45:42.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:45:42.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:45:42.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:45:42.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:45:42.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:45:43.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:45:43.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:45:43.224 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:45:43.224 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:45:43.224 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:43.224 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:45:43.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:45:43.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:45:43.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:45:43.259 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: expr 6 - 1 2024-04-25T08:45:43.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: seq 0 5 2024-04-25T08:45:43.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:45:43.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 0 2024-04-25T08:45:43.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:45:43.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:45:43.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:45:43.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:45:43.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:45:43.266 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:45:43.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:45:43.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:45:43.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:45:43.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:45:43.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:45:43.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:45:43.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:45:43.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:45:43.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4f1428bd-f534-4f73-a7d5-cb7003f29121 2024-04-25T08:45:43.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4f1428bd-f534-4f73-a7d5-cb7003f29121' 2024-04-25T08:45:43.269 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 4f1428bd-f534-4f73-a7d5-cb7003f29121 2024-04-25T08:45:43.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:45:43.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3GCpmLTN7ERAA3FtH93LVWtqoES2teZzlRA== 2024-04-25T08:45:43.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3GCpmLTN7ERAA3FtH93LVWtqoES2teZzlRA=="}' 2024-04-25T08:45:43.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4f1428bd-f534-4f73-a7d5-cb7003f29121 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:45:43.591 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:45:43.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:45:43.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3GCpmLTN7ERAA3FtH93LVWtqoES2teZzlRA== --osd-uuid 4f1428bd-f534-4f73-a7d5-cb7003f29121 2024-04-25T08:45:43.637 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:43.644+0000 7f738e7c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:43.658 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:43.664+0000 7f738e7c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:43.660 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:43.668+0000 7f738e7c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:43.660 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:43.668+0000 7f738e7c37c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:45:43.660 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:43.668+0000 7f738e7c37c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:45:43.660 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:43.668+0000 7f738e7c37c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:45:43.690 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:43.696+0000 7f738e7c37c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:45:49.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:45:49.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:45:49.992 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:45:49.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:45:49.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:45:50.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:45:50.558 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:45:50.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:45:50.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:45:50.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:45:50.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:45:50.590 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:50.596+0000 7fc1e19b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:50.592 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:50.600+0000 7fc1e19b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:50.593 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:50.600+0000 7fc1e19b27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:45:50.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:45:50.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:45:50.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:45:50.985 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:45:50.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:45:50.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:45:50.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:50.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:45:50.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:50.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:45:51.172 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:51.180+0000 7fc1e19b27c0 -1 Falling back to public interface 2024-04-25T08:45:51.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:52.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:52.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:52.418 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:45:52.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:45:52.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:52.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:45:52.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:53.757 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:45:53.764+0000 7fc1e19b27c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:45:53.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:53.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:53.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:45:53.829 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:45:53.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:53.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:45:54.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:55.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:55.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:55.505 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:45:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:45:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:55.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:45:56.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:57.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:57.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:57.006 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:45:57.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:45:57.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:57.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:45:57.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:58.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:58.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:58.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:45:58.412 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:45:58.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:58.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:45:58.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:45:59.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:45:59.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:45:59.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:45:59.927 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:45:59.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:45:59.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:00.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:01.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:01.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:01.380 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:46:01.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:46:01.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:01.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:01.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:02.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:02.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:02.904 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:46:02.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:46:02.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:02.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:03.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:04.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:04.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:04.384 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:46:04.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:46:04.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:04.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:04.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:05.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:05.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:05.940 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:46:05.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:46:05.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:05.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:06.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:07.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:07.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:07.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:46:07.372 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:46:07.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:07.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:08.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:09.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:09.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:09.115 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:46:09.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:46:09.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:09.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:09.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:10.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:10.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:10.867 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:46:10.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:46:10.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:10.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:11.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:12.670 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:46:12.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:12.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:12.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:46:12.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:12.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:13.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:14.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:14.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:14.811 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:46:14.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:46:14.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:14.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:15.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:16.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:16.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:16.591 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:46:16.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:46:16.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:16.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:17.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:18.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:18.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:18.470 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:46:18.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:46:18.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:18.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:46:18.925 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2224943608,v1:127.0.0.1:6803/2224943608] [v2:127.0.0.1:6804/2224943608,v1:127.0.0.1:6805/2224943608] exists,up 4f1428bd-f534-4f73-a7d5-cb7003f29121 2024-04-25T08:46:18.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:46:18.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:46:18.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:46:18.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:46:18.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 1 2024-04-25T08:46:18.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:46:18.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:46:18.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:46:18.929 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:46:18.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:46:18.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:46:18.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:46:18.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:46:18.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1c639d52-e9b6-4a3a-9c35-0fd9e77a7ddb 2024-04-25T08:46:18.931 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 1c639d52-e9b6-4a3a-9c35-0fd9e77a7ddb 2024-04-25T08:46:18.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 1c639d52-e9b6-4a3a-9c35-0fd9e77a7ddb' 2024-04-25T08:46:18.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:46:18.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaGCpmY/BmORAAHgu6505lzEWOG5jaTlX7oQ== 2024-04-25T08:46:18.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaGCpmY/BmORAAHgu6505lzEWOG5jaTlX7oQ=="}' 2024-04-25T08:46:18.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1c639d52-e9b6-4a3a-9c35-0fd9e77a7ddb -i td/osd-recovery-stats/1/new.json 2024-04-25T08:46:19.487 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:46:19.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:46:19.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaGCpmY/BmORAAHgu6505lzEWOG5jaTlX7oQ== --osd-uuid 1c639d52-e9b6-4a3a-9c35-0fd9e77a7ddb 2024-04-25T08:46:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:19.543+0000 7f6dc51617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:19.554 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:19.559+0000 7f6dc51617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:19.556 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:19.563+0000 7f6dc51617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:19.556 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:19.563+0000 7f6dc51617c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:46:19.556 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:19.563+0000 7f6dc51617c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:46:19.556 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:19.563+0000 7f6dc51617c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:46:19.593 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:19.599+0000 7f6dc51617c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:46:26.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:46:26.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:46:26.204 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:46:26.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:46:26.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:46:26.783 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:46:26.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:46:26.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:46:26.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:46:26.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:46:26.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:46:26.815 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:26.823+0000 7f5e71a9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:26.817 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:26.823+0000 7f5e71a9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:26.818 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:26.827+0000 7f5e71a9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:27.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:46:27.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:46:27.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:46:27.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:46:27.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:46:27.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:27.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:46:27.171 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:46:27.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:27.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:27.361 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:27.367+0000 7f5e71a9c7c0 -1 Falling back to public interface 2024-04-25T08:46:27.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:28.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:28.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:28.638 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:46:28.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:46:28.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:28.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:29.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:30.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:30.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:30.040 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:46:30.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:46:30.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:30.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:30.055 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:30.063+0000 7f5e71a9c7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:46:30.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:31.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:31.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:31.713 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:46:31.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:46:31.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:31.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:32.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:33.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:33.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:33.318 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:46:33.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:46:33.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:33.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:33.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:34.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:34.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:34.798 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:46:34.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:46:34.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:34.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:35.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:36.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:36.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:36.260 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:46:36.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:46:36.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:36.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:36.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:37.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:37.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:37.775 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:46:37.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:46:37.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:37.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:38.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:39.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:39.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:39.280 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:46:39.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:46:39.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:39.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:39.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:40.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:40.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:40.747 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:46:40.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:46:40.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:40.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:41.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:42.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:42.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:42.253 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:46:42.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:46:42.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:42.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:42.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:43.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:43.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:43.780 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:46:43.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:46:43.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:43.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:44.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:45.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:45.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:45.609 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:46:45.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:46:45.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:45.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:46.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:47.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:47.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:47.219 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:46:47.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:46:47.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:47.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:48.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:49.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:49.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:49.067 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:46:49.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:46:49.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:49.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:49.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:50.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:50.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:50.922 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:46:50.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:46:50.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:50.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:51.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:52.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:52.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:52.724 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:46:52.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:46:52.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:52.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:53.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:54.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:54.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:54.968 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:46:54.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:46:54.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:54.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:55.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:46:56.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:46:56.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:46:56.582 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:46:56.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:46:56.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:46:56.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:46:57.011 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2698341378,v1:127.0.0.1:6811/2698341378] [v2:127.0.0.1:6812/2698341378,v1:127.0.0.1:6813/2698341378] exists,up 1c639d52-e9b6-4a3a-9c35-0fd9e77a7ddb 2024-04-25T08:46:57.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:46:57.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:46:57.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:46:57.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:46:57.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 2 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:46:57.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:46:57.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:46:57.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:46:57.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:46:57.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:46:57.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:46:57.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:46:57.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:46:57.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b0e0c852-2143-4c51-82f9-2de29c2d38c7 2024-04-25T08:46:57.019 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 b0e0c852-2143-4c51-82f9-2de29c2d38c7 2024-04-25T08:46:57.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b0e0c852-2143-4c51-82f9-2de29c2d38c7' 2024-04-25T08:46:57.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:46:57.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCBGCpmHYP+AhAAzoPH9JYhX3BPzC7FmfgmRg== 2024-04-25T08:46:57.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCBGCpmHYP+AhAAzoPH9JYhX3BPzC7FmfgmRg=="}' 2024-04-25T08:46:57.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b0e0c852-2143-4c51-82f9-2de29c2d38c7 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:46:57.656 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:46:57.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:46:57.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBGCpmHYP+AhAAzoPH9JYhX3BPzC7FmfgmRg== --osd-uuid b0e0c852-2143-4c51-82f9-2de29c2d38c7 2024-04-25T08:46:57.703 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:57.710+0000 7f93d81407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:57.709 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:57.714+0000 7f93d81407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:57.710 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:57.714+0000 7f93d81407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:46:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:57.718+0000 7f93d81407c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:46:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:57.718+0000 7f93d81407c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:46:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:57.718+0000 7f93d81407c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:46:57.748 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:46:57.754+0000 7f93d81407c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:47:04.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:47:04.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:47:04.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:47:04.391 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T08:47:04.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:47:04.948 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T08:47:04.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:47:04.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:47:04.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:47:04.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:47:04.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:47:04.980 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:04.986+0000 7f9df75a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:04.982 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:04.990+0000 7f9df75a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:04.984 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:04.990+0000 7f9df75a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:05.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:47:05.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:47:05.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:47:05.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:47:05.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:47:05.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:05.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:47:05.358 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:47:05.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:05.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:05.548 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:05.554+0000 7f9df75a47c0 -1 Falling back to public interface 2024-04-25T08:47:05.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:06.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:06.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:06.793 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:47:06.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:47:06.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:06.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:07.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:08.171 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:08.178+0000 7f9df75a47c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:47:08.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:08.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:08.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:47:08.202 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:47:08.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:08.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:08.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:09.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:09.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:09.935 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:47:09.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:47:09.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:09.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:10.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:11.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:11.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:11.410 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:47:11.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:47:11.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:11.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:11.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:12.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:12.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:12.837 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:47:12.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:47:12.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:12.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:13.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:14.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:14.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:14.283 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:47:14.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:47:14.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:14.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:14.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:15.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:15.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:15.813 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:47:15.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:47:15.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:15.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:16.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:17.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:17.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:17.604 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:47:17.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:47:17.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:17.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:18.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:19.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:19.289 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:47:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:47:19.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:19.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:19.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:20.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:20.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:20.718 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:47:20.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:47:20.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:20.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:21.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:22.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:22.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:22.208 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:47:22.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:47:22.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:22.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:22.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:23.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:23.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:23.892 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:47:23.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:47:23.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:23.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:24.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:25.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:25.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:25.998 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:47:25.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:47:25.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:25.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:26.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:27.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:27.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:27.982 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:47:27.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:47:27.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:27.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:28.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:29.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:29.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:29.839 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:47:29.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:47:29.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:29.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:31.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:32.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:32.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:32.347 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:47:32.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:47:32.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:32.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:33.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:34.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:34.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:34.301 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:47:34.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:47:34.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:34.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:47:34.784 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/966467605,v1:127.0.0.1:6819/966467605] [v2:127.0.0.1:6820/966467605,v1:127.0.0.1:6821/966467605] exists,up b0e0c852-2143-4c51-82f9-2de29c2d38c7 2024-04-25T08:47:34.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:47:34.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:47:34.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 3 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:47:34.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:47:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:47:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:47:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:47:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:47:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:47:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:47:34.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:47:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9fc5afcd-8d97-4fcf-b1dc-6de08612482d 2024-04-25T08:47:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 9fc5afcd-8d97-4fcf-b1dc-6de08612482d' 2024-04-25T08:47:34.790 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 9fc5afcd-8d97-4fcf-b1dc-6de08612482d 2024-04-25T08:47:34.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:47:34.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmGCpmNNvkMBAAWcWz267z0z+9o7QjNUuRFw== 2024-04-25T08:47:34.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmGCpmNNvkMBAAWcWz267z0z+9o7QjNUuRFw=="}' 2024-04-25T08:47:34.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9fc5afcd-8d97-4fcf-b1dc-6de08612482d -i td/osd-recovery-stats/3/new.json 2024-04-25T08:47:35.481 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:47:35.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:47:35.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmGCpmNNvkMBAAWcWz267z0z+9o7QjNUuRFw== --osd-uuid 9fc5afcd-8d97-4fcf-b1dc-6de08612482d 2024-04-25T08:47:35.536 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:35.545+0000 7f34621c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:35.545 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:35.553+0000 7f34621c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:35.547 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:35.557+0000 7f34621c97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:35.547 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:35.557+0000 7f34621c97c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:47:35.547 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:35.557+0000 7f34621c97c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:47:35.547 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:35.557+0000 7f34621c97c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:47:35.620 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:35.629+0000 7f34621c97c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:47:42.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:47:42.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:47:42.593 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T08:47:42.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:47:42.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:47:43.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:47:43.173 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T08:47:43.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:47:43.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:47:43.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:47:43.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:47:43.204 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:43.213+0000 7f7508e957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:43.206 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:43.213+0000 7f7508e957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:43.207 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:43.217+0000 7f7508e957c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:47:43.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:47:43.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:47:43.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:47:43.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:47:43.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:47:43.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:43.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:47:43.570 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:47:43.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:43.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:43.757 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:43.765+0000 7f7508e957c0 -1 Falling back to public interface 2024-04-25T08:47:44.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:45.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:45.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:45.022 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:47:45.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:47:45.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:45.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:45.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:46.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:46.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:46.436 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:47:46.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:47:46.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:46.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:46.475 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:47:46.485+0000 7f7508e957c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:47:46.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:47.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:47.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:47.905 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:47:47.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:47:47.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:47.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:48.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:49.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:49.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:49.462 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:47:49.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:47:49.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:49.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:49.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:50.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:50.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:50.901 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:47:50.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:47:50.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:50.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:51.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:52.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:52.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:52.459 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:47:52.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:47:52.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:52.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:52.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:53.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:53.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:53.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:47:53.911 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:47:53.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:53.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:54.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:55.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:55.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:55.450 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:47:55.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:47:55.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:55.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:55.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:56.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:56.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:56.896 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:47:56.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:47:56.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:56.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:57.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:58.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:58.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:58.426 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:47:58.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:47:58.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:58.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:47:58.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:47:59.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:47:59.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:47:59.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:47:59.931 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:47:59.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:47:59.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:00.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:01.392 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:48:01.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:48:01.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:01.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:02.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:03.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:03.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:03.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:48:03.179 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:48:03.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:03.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:04.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:05.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:05.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:05.095 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:48:05.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:48:05.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:05.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:05.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:06.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:06.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:06.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:48:06.917 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:48:06.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:06.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:07.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:08.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:08.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:08.677 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:48:08.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:48:08.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:08.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:09.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:10.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:10.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:10.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:48:10.875 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:48:10.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:10.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:11.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:12.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:12.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:12.796 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:48:12.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:48:12.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:12.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:48:13.224 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1407172160,v1:127.0.0.1:6827/1407172160] [v2:127.0.0.1:6828/1407172160,v1:127.0.0.1:6829/1407172160] exists,up 9fc5afcd-8d97-4fcf-b1dc-6de08612482d 2024-04-25T08:48:13.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:48:13.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:48:13.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:48:13.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:48:13.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 4 2024-04-25T08:48:13.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:48:13.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:48:13.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:48:13.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:48:13.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:48:13.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:48:13.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:48:13.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=526eea77-eacc-4086-8216-0ae5f0978396 2024-04-25T08:48:13.230 INFO:tasks.workunit.client.0.smithi134.stdout:add osd4 526eea77-eacc-4086-8216-0ae5f0978396 2024-04-25T08:48:13.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 526eea77-eacc-4086-8216-0ae5f0978396' 2024-04-25T08:48:13.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:48:13.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDNGCpmYfA5DxAATkn8sM9fRQfhRp0X5+uiiQ== 2024-04-25T08:48:13.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDNGCpmYfA5DxAATkn8sM9fRQfhRp0X5+uiiQ=="}' 2024-04-25T08:48:13.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 526eea77-eacc-4086-8216-0ae5f0978396 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:48:13.745 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:48:13.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:48:13.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDNGCpmYfA5DxAATkn8sM9fRQfhRp0X5+uiiQ== --osd-uuid 526eea77-eacc-4086-8216-0ae5f0978396 2024-04-25T08:48:13.790 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:13.796+0000 7f131ba6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:13.792 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:13.800+0000 7f131ba6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:13.793 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:13.800+0000 7f131ba6f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:13.793 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:13.800+0000 7f131ba6f7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:48:13.794 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:13.800+0000 7f131ba6f7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:48:13.794 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:13.800+0000 7f131ba6f7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:48:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:13.872+0000 7f131ba6f7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:48:21.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:48:21.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:48:21.177 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd4 key to auth repository 2024-04-25T08:48:21.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:48:21.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:48:21.737 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.4 2024-04-25T08:48:21.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:48:21.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:48:21.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:48:21.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:48:21.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:48:21.768 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:21.776+0000 7f75cd82a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:21.797 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:21.804+0000 7f75cd82a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:21.798 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:21.804+0000 7f75cd82a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:22.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:48:22.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:48:22.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:48:22.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:48:22.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:48:22.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:22.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:48:22.200 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:48:22.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:22.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:22.388 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:22.396+0000 7f75cd82a7c0 -1 Falling back to public interface 2024-04-25T08:48:22.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:23.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:23.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:23.652 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:48:23.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:48:23.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:23.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:24.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:25.015 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:25.020+0000 7f75cd82a7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:48:25.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:25.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:25.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:48:25.092 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:48:25.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:25.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:25.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:26.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:26.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:26.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:48:26.768 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:48:26.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:26.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:27.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:28.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:28.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:28.349 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:48:28.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:48:28.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:28.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:28.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:29.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:29.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:29.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:48:29.874 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:48:29.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:29.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:30.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:31.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:31.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:31.394 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:48:31.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:48:31.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:31.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:31.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:32.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:32.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:32.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:48:32.919 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:48:32.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:32.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:33.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:34.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:34.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:34.347 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:48:34.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:48:34.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:34.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:34.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:35.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:35.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:35.795 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:48:35.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:48:35.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:35.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:36.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:37.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:37.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:37.264 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:48:37.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:48:37.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:37.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:37.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:38.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:38.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:38.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:48:38.695 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:48:38.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:38.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:39.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:40.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:40.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:40.270 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:48:40.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:48:40.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:40.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:41.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:42.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:42.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:42.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:48:42.573 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:48:42.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:42.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:43.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:44.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:44.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:44.375 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:48:44.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:48:44.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:44.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:45.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:46.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:46.133 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:48:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:48:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:47.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:48.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:48.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:48.184 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:48:48.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:48:48.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:48.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:49.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:50.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:50.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:50.436 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:48:50.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:48:50.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:50.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:51.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:48:52.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:48:52.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:48:52.274 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:48:52.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:48:52.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:48:52.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:48:52.738 INFO:tasks.workunit.client.0.smithi134.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/584507607,v1:127.0.0.1:6835/584507607] [v2:127.0.0.1:6836/584507607,v1:127.0.0.1:6837/584507607] exists,up 526eea77-eacc-4086-8216-0ae5f0978396 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:446: TEST_recovery_multi: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:448: TEST_recovery_multi: run_osd td/osd-recovery-stats 5 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:48:52.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:48:52.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:48:52.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ca9af35-f291-4218-b2e2-397210b9f6e2 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 4ca9af35-f291-4218-b2e2-397210b9f6e2' 2024-04-25T08:48:52.746 INFO:tasks.workunit.client.0.smithi134.stdout:add osd5 4ca9af35-f291-4218-b2e2-397210b9f6e2 2024-04-25T08:48:52.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:48:52.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD0GCpm3ErHLRAAyguYFItk7PI4Ot8LYmtW0Q== 2024-04-25T08:48:52.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD0GCpm3ErHLRAAyguYFItk7PI4Ot8LYmtW0Q=="}' 2024-04-25T08:48:52.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ca9af35-f291-4218-b2e2-397210b9f6e2 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:48:53.281 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:48:53.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:48:53.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD0GCpm3ErHLRAAyguYFItk7PI4Ot8LYmtW0Q== --osd-uuid 4ca9af35-f291-4218-b2e2-397210b9f6e2 2024-04-25T08:48:53.333 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:53.339+0000 7f0429a5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:53.358 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:53.363+0000 7f0429a5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:53.359 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:53.367+0000 7f0429a5e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:48:53.360 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:53.367+0000 7f0429a5e7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:48:53.360 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:53.367+0000 7f0429a5e7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:48:53.360 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:53.367+0000 7f0429a5e7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:48:53.423 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:48:53.431+0000 7f0429a5e7c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:49:00.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:49:00.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:49:00.322 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd5 key to auth repository 2024-04-25T08:49:00.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:49:00.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:49:00.887 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.5 2024-04-25T08:49:00.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:49:00.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:49:00.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:49:00.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:49:00.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:49:00.918 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:49:00.923+0000 7fcaa9cd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:00.959 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:49:00.967+0000 7fcaa9cd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:00.960 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:49:00.967+0000 7fcaa9cd47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:01.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:49:01.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:49:01.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:49:01.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:49:01.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:49:01.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:01.306 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:49:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:49:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:01.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:01.513 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:49:01.519+0000 7fcaa9cd47c0 -1 Falling back to public interface 2024-04-25T08:49:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:02.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:02.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:02.750 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:49:02.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:49:02.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:02.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:03.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:04.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:04.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:04.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:49:04.186 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:49:04.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:04.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:04.220 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:49:04.227+0000 7fcaa9cd47c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:49:04.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:05.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:05.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:05.809 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:49:05.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:49:05.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:05.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:06.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:07.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:07.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:07.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:49:07.405 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:49:07.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:07.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:07.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:08.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:08.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:08.857 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:49:08.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:49:08.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:08.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:09.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:10.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:10.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:10.435 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:49:10.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:49:10.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:10.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:10.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:11.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:11.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:11.874 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:49:11.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:49:11.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:11.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:12.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:13.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:13.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:13.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:49:13.424 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:49:13.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:13.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:13.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:14.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:14.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:14.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:49:14.906 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:49:14.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:14.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:15.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:16.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:16.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:16.384 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:49:16.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:49:16.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:16.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:16.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:17.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:17.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:17.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:49:17.824 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:49:17.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:17.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:18.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:19.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:19.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:19.364 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:49:19.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:49:19.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:19.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:20.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:21.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:21.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:21.381 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:49:21.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:49:21.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:21.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:22.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:23.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:23.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:23.060 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:49:23.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:49:23.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:23.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:23.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:24.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:24.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:24.805 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:49:24.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:49:24.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:24.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:25.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:26.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:26.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:26.530 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:49:26.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:49:26.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:26.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:27.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:28.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:28.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:28.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:49:28.251 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:49:28.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:28.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:28.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:49:29.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:49:29.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:29.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:49:29.980 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:49:29.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:29.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:49:30.489 INFO:tasks.workunit.client.0.smithi134.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3210820165,v1:127.0.0.1:6843/3210820165] [v2:127.0.0.1:6844/3210820165,v1:127.0.0.1:6845/3210820165] exists,up 4ca9af35-f291-4218-b2e2-397210b9f6e2 2024-04-25T08:49:30.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:49:30.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:49:30.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:49:30.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:451: TEST_recovery_multi: create_pool test 1 1 2024-04-25T08:49:30.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:49:31.279 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T08:49:31.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:49:32.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:452: TEST_recovery_multi: ceph osd pool set test size 3 2024-04-25T08:49:33.149 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 3 2024-04-25T08:49:33.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:453: TEST_recovery_multi: ceph osd pool set test min_size 1 2024-04-25T08:49:34.416 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 min_size to 1 2024-04-25T08:49:34.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:455: TEST_recovery_multi: wait_for_clean 2024-04-25T08:49:34.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:49:34.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:49:34.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:49:34.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:49:34.443 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:49:34.443 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:49:34.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:49:34.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:49:34.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:49:34.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:49:34.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:49:34.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:49:34.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:49:34.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:49:34.561 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:49:35.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:49:35.186 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:49:35.186 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:49:35.187 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:49:35.187 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T08:49:35.187 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T08:49:35.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:49:35.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:35.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:49:35.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836527 2024-04-25T08:49:35.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836527 2024-04-25T08:49:35.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527' 2024-04-25T08:49:35.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:35.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:49:35.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672996 2024-04-25T08:49:35.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672996 2024-04-25T08:49:35.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672996' 2024-04-25T08:49:35.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:35.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:49:35.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509476 2024-04-25T08:49:35.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509476 2024-04-25T08:49:35.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672996 2-64424509476' 2024-04-25T08:49:35.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:35.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:49:35.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345943 2024-04-25T08:49:35.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345943 2024-04-25T08:49:35.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672996 2-64424509476 3-85899345943' 2024-04-25T08:49:35.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:35.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:49:35.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182414 2024-04-25T08:49:35.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182414 2024-04-25T08:49:35.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672996 2-64424509476 3-85899345943 4-107374182414' 2024-04-25T08:49:35.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:49:35.987 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:49:36.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018892 2024-04-25T08:49:36.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018892 2024-04-25T08:49:36.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672996 2-64424509476 3-85899345943 4-107374182414 5-128849018892' 2024-04-25T08:49:36.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:36.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836527 2024-04-25T08:49:36.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:36.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:49:36.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836527 2024-04-25T08:49:36.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:36.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836527 2024-04-25T08:49:36.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-04-25T08:49:36.141 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836527 2024-04-25T08:49:36.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:49:36.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836527 -lt 21474836527 2024-04-25T08:49:36.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:36.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672996 2024-04-25T08:49:36.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:36.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:49:36.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672996 2024-04-25T08:49:36.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:36.573 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672996 2024-04-25T08:49:36.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672996 2024-04-25T08:49:36.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-04-25T08:49:36.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:49:37.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672996 -lt 42949672996 2024-04-25T08:49:37.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:37.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509476 2024-04-25T08:49:37.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:37.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:49:37.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509476 2024-04-25T08:49:37.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:37.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509476 2024-04-25T08:49:37.037 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509476 2024-04-25T08:49:37.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509476' 2024-04-25T08:49:37.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:49:37.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509477 -lt 64424509476 2024-04-25T08:49:37.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:37.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:37.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345943 2024-04-25T08:49:37.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:49:37.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345943 2024-04-25T08:49:37.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:37.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345943 2024-04-25T08:49:37.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345943' 2024-04-25T08:49:37.491 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345943 2024-04-25T08:49:37.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:49:37.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345943 -lt 85899345943 2024-04-25T08:49:37.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:37.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182414 2024-04-25T08:49:37.898 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:37.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:49:37.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182414 2024-04-25T08:49:37.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:37.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182414 2024-04-25T08:49:37.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182414' 2024-04-25T08:49:37.902 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182414 2024-04-25T08:49:37.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:49:38.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182415 -lt 107374182414 2024-04-25T08:49:38.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:49:38.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:49:38.311 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018892 2024-04-25T08:49:38.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:49:38.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018892 2024-04-25T08:49:38.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:49:38.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018892 2024-04-25T08:49:38.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018892' 2024-04-25T08:49:38.314 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018892 2024-04-25T08:49:38.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:49:38.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018892 -lt 128849018892 2024-04-25T08:49:38.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:49:38.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:38.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:39.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:49:39.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:49:39.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:49:39.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:49:39.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:49:39.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:49:39.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:49:39.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:49:39.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:49:39.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:49:39.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:49:39.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:49:40.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:49:40.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:49:40.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:49:40.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:457: TEST_recovery_multi: rados -p test put obj1 /dev/null 2024-04-25T08:49:40.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:49:40.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:49:40.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:49:40.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:49:40.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:49:40.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:459: TEST_recovery_multi: local primary=1 2024-04-25T08:49:40.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: get_not_primary test obj1 2024-04-25T08:49:40.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:49:40.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:49:40.798 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:49:40.798 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:49:40.798 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:49:40.798 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:49:40.798 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:49:41.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:49:41.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:49:41.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:49:41.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:460: TEST_recovery_multi: local otherosd=0 2024-04-25T08:49:41.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:462: TEST_recovery_multi: ceph osd set noout 2024-04-25T08:49:42.456 INFO:tasks.workunit.client.0.smithi134.stderr:noout is set 2024-04-25T08:49:42.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:463: TEST_recovery_multi: ceph osd set norecover 2024-04-25T08:49:43.268 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is set 2024-04-25T08:49:43.292 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:49:43.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:464: TEST_recovery_multi: kill 310839 2024-04-25T08:49:43.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:465: TEST_recovery_multi: ceph osd down osd.0 2024-04-25T08:49:43.802 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T08:49:43.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: expr 200 / 2 2024-04-25T08:49:43.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:467: TEST_recovery_multi: local half=100 2024-04-25T08:49:43.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: seq 2 100 2024-04-25T08:49:43.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:43.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj2 /dev/null 2024-04-25T08:49:44.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:44.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj3 /dev/null 2024-04-25T08:49:44.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:44.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj4 /dev/null 2024-04-25T08:49:45.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:45.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj5 /dev/null 2024-04-25T08:49:45.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:45.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj6 /dev/null 2024-04-25T08:49:45.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:45.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj7 /dev/null 2024-04-25T08:49:45.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:45.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj8 /dev/null 2024-04-25T08:49:45.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:45.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj9 /dev/null 2024-04-25T08:49:45.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:45.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj10 /dev/null 2024-04-25T08:49:45.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:45.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj11 /dev/null 2024-04-25T08:49:46.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:46.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj12 /dev/null 2024-04-25T08:49:46.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:46.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj13 /dev/null 2024-04-25T08:49:46.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:46.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj14 /dev/null 2024-04-25T08:49:46.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:46.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj15 /dev/null 2024-04-25T08:49:46.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:46.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj16 /dev/null 2024-04-25T08:49:46.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:46.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj17 /dev/null 2024-04-25T08:49:46.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:46.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj18 /dev/null 2024-04-25T08:49:46.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:46.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj19 /dev/null 2024-04-25T08:49:47.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:47.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj20 /dev/null 2024-04-25T08:49:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:47.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj21 /dev/null 2024-04-25T08:49:47.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:47.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj22 /dev/null 2024-04-25T08:49:47.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:47.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj23 /dev/null 2024-04-25T08:49:47.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:47.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj24 /dev/null 2024-04-25T08:49:47.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:47.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj25 /dev/null 2024-04-25T08:49:47.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:47.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj26 /dev/null 2024-04-25T08:49:48.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:48.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj27 /dev/null 2024-04-25T08:49:48.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:48.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj28 /dev/null 2024-04-25T08:49:48.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:48.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj29 /dev/null 2024-04-25T08:49:48.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:48.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj30 /dev/null 2024-04-25T08:49:48.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:48.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj31 /dev/null 2024-04-25T08:49:48.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:48.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj32 /dev/null 2024-04-25T08:49:49.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:49.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj33 /dev/null 2024-04-25T08:49:49.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:49.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj34 /dev/null 2024-04-25T08:49:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:49.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj35 /dev/null 2024-04-25T08:49:49.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:49.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj36 /dev/null 2024-04-25T08:49:49.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:49.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj37 /dev/null 2024-04-25T08:49:49.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:49.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj38 /dev/null 2024-04-25T08:49:49.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:49.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj39 /dev/null 2024-04-25T08:49:49.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:49.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj40 /dev/null 2024-04-25T08:49:50.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:50.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj41 /dev/null 2024-04-25T08:49:50.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:50.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj42 /dev/null 2024-04-25T08:49:50.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:50.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj43 /dev/null 2024-04-25T08:49:50.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:50.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj44 /dev/null 2024-04-25T08:49:50.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:50.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj45 /dev/null 2024-04-25T08:49:50.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:50.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj46 /dev/null 2024-04-25T08:49:50.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:50.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj47 /dev/null 2024-04-25T08:49:50.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:50.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj48 /dev/null 2024-04-25T08:49:51.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:51.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj49 /dev/null 2024-04-25T08:49:51.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:51.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj50 /dev/null 2024-04-25T08:49:51.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:51.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj51 /dev/null 2024-04-25T08:49:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj52 /dev/null 2024-04-25T08:49:51.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:51.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj53 /dev/null 2024-04-25T08:49:51.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:51.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj54 /dev/null 2024-04-25T08:49:51.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:51.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj55 /dev/null 2024-04-25T08:49:52.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:52.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj56 /dev/null 2024-04-25T08:49:52.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:52.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj57 /dev/null 2024-04-25T08:49:52.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:52.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj58 /dev/null 2024-04-25T08:49:52.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:52.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj59 /dev/null 2024-04-25T08:49:52.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:52.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj60 /dev/null 2024-04-25T08:49:52.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:52.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj61 /dev/null 2024-04-25T08:49:52.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:52.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj62 /dev/null 2024-04-25T08:49:52.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:52.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj63 /dev/null 2024-04-25T08:49:53.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:53.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj64 /dev/null 2024-04-25T08:49:53.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:53.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj65 /dev/null 2024-04-25T08:49:53.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:53.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj66 /dev/null 2024-04-25T08:49:53.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:53.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj67 /dev/null 2024-04-25T08:49:53.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:53.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj68 /dev/null 2024-04-25T08:49:53.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:53.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj69 /dev/null 2024-04-25T08:49:53.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:53.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj70 /dev/null 2024-04-25T08:49:54.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:54.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj71 /dev/null 2024-04-25T08:49:54.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:54.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj72 /dev/null 2024-04-25T08:49:54.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:54.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj73 /dev/null 2024-04-25T08:49:54.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:54.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj74 /dev/null 2024-04-25T08:49:54.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:54.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj75 /dev/null 2024-04-25T08:49:54.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:54.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj76 /dev/null 2024-04-25T08:49:54.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:54.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj77 /dev/null 2024-04-25T08:49:55.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:55.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj78 /dev/null 2024-04-25T08:49:55.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:55.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj79 /dev/null 2024-04-25T08:49:55.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:55.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj80 /dev/null 2024-04-25T08:49:55.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:55.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj81 /dev/null 2024-04-25T08:49:55.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:55.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj82 /dev/null 2024-04-25T08:49:55.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:55.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj83 /dev/null 2024-04-25T08:49:55.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:55.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj84 /dev/null 2024-04-25T08:49:55.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:55.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj85 /dev/null 2024-04-25T08:49:56.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:56.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj86 /dev/null 2024-04-25T08:49:56.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:56.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj87 /dev/null 2024-04-25T08:49:56.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:56.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj88 /dev/null 2024-04-25T08:49:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj89 /dev/null 2024-04-25T08:49:56.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:56.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj90 /dev/null 2024-04-25T08:49:56.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:56.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj91 /dev/null 2024-04-25T08:49:56.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:56.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj92 /dev/null 2024-04-25T08:49:56.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj93 /dev/null 2024-04-25T08:49:57.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:57.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj94 /dev/null 2024-04-25T08:49:57.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:57.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj95 /dev/null 2024-04-25T08:49:57.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:57.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj96 /dev/null 2024-04-25T08:49:57.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:57.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj97 /dev/null 2024-04-25T08:49:57.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:57.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj98 /dev/null 2024-04-25T08:49:57.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:57.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj99 /dev/null 2024-04-25T08:49:57.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:468: TEST_recovery_multi: for i in $(seq 2 $half) 2024-04-25T08:49:57.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:470: TEST_recovery_multi: rados -p test put obj100 /dev/null 2024-04-25T08:49:57.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: cat td/osd-recovery-stats/osd.1.pid 2024-04-25T08:49:57.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:473: TEST_recovery_multi: kill 312577 2024-04-25T08:49:57.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:474: TEST_recovery_multi: ceph osd down osd.1 2024-04-25T08:49:58.808 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already down. 2024-04-25T08:49:58.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:475: TEST_recovery_multi: activate_osd td/osd-recovery-stats 0 2024-04-25T08:49:58.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-04-25T08:49:58.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:49:58.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T08:49:58.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:49:58.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:49:58.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:49:58.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:49:58.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:49:58.834 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:49:58.834 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:49:58.834 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:49:58.834 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:49:58.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:49:58.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:49:58.835 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:49:58.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T08:49:58.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:49:58.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/0/whoami 2024-04-25T08:49:58.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T08:49:58.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:49:58.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:49:58.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:49:58.868 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:49:58.878+0000 7f526ec007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:58.890 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:49:58.898+0000 7f526ec007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:58.891 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:49:58.902+0000 7f526ec007c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:49:59.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T08:49:59.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:49:59.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:49:59.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:49:59.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:49:59.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:49:59.391 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:49:59.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:49:59.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:49:59.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:49:59.449 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:49:59.458+0000 7f526ec007c0 -1 Falling back to public interface 2024-04-25T08:49:59.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:00.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:00.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:00.832 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:50:00.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:50:00.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:00.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:50:01.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:02.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:02.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:02.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:50:02.252 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:50:02.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:02.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:50:02.556 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:50:02.565+0000 7f526ec007c0 -1 osd.0 43 log_to_monitors true 2024-04-25T08:50:02.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:03.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:03.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:03.808 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:50:03.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:50:03.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:03.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:50:04.229 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 49 up_thru 49 down_at 44 last_clean_interval [5,43) [v2:127.0.0.1:6802/599398886,v1:127.0.0.1:6803/599398886] [v2:127.0.0.1:6804/599398886,v1:127.0.0.1:6805/599398886] exists,up 4f1428bd-f534-4f73-a7d5-cb7003f29121 2024-04-25T08:50:04.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:50:04.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:50:04.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:50:04.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:476: TEST_recovery_multi: sleep 3 2024-04-25T08:50:07.233 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: expr 100 + 1 2024-04-25T08:50:07.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: seq 101 200 2024-04-25T08:50:07.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:07.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj101 /dev/null 2024-04-25T08:50:07.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:07.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj102 /dev/null 2024-04-25T08:50:07.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:07.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj103 /dev/null 2024-04-25T08:50:07.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:07.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj104 /dev/null 2024-04-25T08:50:07.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:07.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj105 /dev/null 2024-04-25T08:50:07.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:07.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj106 /dev/null 2024-04-25T08:50:07.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:07.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj107 /dev/null 2024-04-25T08:50:08.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:08.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj108 /dev/null 2024-04-25T08:50:08.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:08.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj109 /dev/null 2024-04-25T08:50:08.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:08.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj110 /dev/null 2024-04-25T08:50:08.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:08.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj111 /dev/null 2024-04-25T08:50:08.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:08.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj112 /dev/null 2024-04-25T08:50:08.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:08.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj113 /dev/null 2024-04-25T08:50:08.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:08.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj114 /dev/null 2024-04-25T08:50:08.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:08.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj115 /dev/null 2024-04-25T08:50:09.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:09.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj116 /dev/null 2024-04-25T08:50:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:09.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj117 /dev/null 2024-04-25T08:50:09.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:09.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj118 /dev/null 2024-04-25T08:50:09.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:09.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj119 /dev/null 2024-04-25T08:50:09.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:09.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj120 /dev/null 2024-04-25T08:50:09.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:09.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj121 /dev/null 2024-04-25T08:50:09.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:09.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj122 /dev/null 2024-04-25T08:50:09.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:09.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj123 /dev/null 2024-04-25T08:50:10.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:10.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj124 /dev/null 2024-04-25T08:50:10.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:10.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj125 /dev/null 2024-04-25T08:50:10.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:10.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj126 /dev/null 2024-04-25T08:50:10.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:10.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj127 /dev/null 2024-04-25T08:50:10.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:10.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj128 /dev/null 2024-04-25T08:50:10.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:10.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj129 /dev/null 2024-04-25T08:50:10.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:10.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj130 /dev/null 2024-04-25T08:50:10.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:10.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj131 /dev/null 2024-04-25T08:50:10.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:10.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj132 /dev/null 2024-04-25T08:50:11.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:11.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj133 /dev/null 2024-04-25T08:50:11.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:11.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj134 /dev/null 2024-04-25T08:50:11.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:11.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj135 /dev/null 2024-04-25T08:50:11.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:11.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj136 /dev/null 2024-04-25T08:50:11.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:11.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj137 /dev/null 2024-04-25T08:50:11.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:11.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj138 /dev/null 2024-04-25T08:50:11.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:11.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj139 /dev/null 2024-04-25T08:50:11.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:11.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj140 /dev/null 2024-04-25T08:50:12.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:12.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj141 /dev/null 2024-04-25T08:50:12.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:12.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj142 /dev/null 2024-04-25T08:50:12.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:12.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj143 /dev/null 2024-04-25T08:50:12.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:12.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj144 /dev/null 2024-04-25T08:50:12.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:12.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj145 /dev/null 2024-04-25T08:50:12.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:12.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj146 /dev/null 2024-04-25T08:50:12.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:12.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj147 /dev/null 2024-04-25T08:50:12.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:12.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj148 /dev/null 2024-04-25T08:50:12.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:12.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj149 /dev/null 2024-04-25T08:50:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj150 /dev/null 2024-04-25T08:50:13.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:13.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj151 /dev/null 2024-04-25T08:50:13.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:13.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj152 /dev/null 2024-04-25T08:50:13.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:13.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj153 /dev/null 2024-04-25T08:50:13.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:13.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj154 /dev/null 2024-04-25T08:50:13.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:13.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj155 /dev/null 2024-04-25T08:50:13.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:13.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj156 /dev/null 2024-04-25T08:50:13.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:13.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj157 /dev/null 2024-04-25T08:50:14.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:14.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj158 /dev/null 2024-04-25T08:50:14.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:14.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj159 /dev/null 2024-04-25T08:50:14.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:14.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj160 /dev/null 2024-04-25T08:50:14.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:14.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj161 /dev/null 2024-04-25T08:50:14.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:14.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj162 /dev/null 2024-04-25T08:50:14.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:14.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj163 /dev/null 2024-04-25T08:50:14.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:14.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj164 /dev/null 2024-04-25T08:50:15.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:15.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj165 /dev/null 2024-04-25T08:50:15.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:15.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj166 /dev/null 2024-04-25T08:50:15.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:15.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj167 /dev/null 2024-04-25T08:50:15.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:15.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj168 /dev/null 2024-04-25T08:50:15.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:15.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj169 /dev/null 2024-04-25T08:50:15.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:15.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj170 /dev/null 2024-04-25T08:50:15.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:15.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj171 /dev/null 2024-04-25T08:50:15.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:15.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj172 /dev/null 2024-04-25T08:50:15.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:15.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj173 /dev/null 2024-04-25T08:50:16.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:16.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj174 /dev/null 2024-04-25T08:50:16.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:16.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj175 /dev/null 2024-04-25T08:50:16.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:16.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj176 /dev/null 2024-04-25T08:50:16.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:16.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj177 /dev/null 2024-04-25T08:50:16.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:16.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj178 /dev/null 2024-04-25T08:50:16.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:16.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj179 /dev/null 2024-04-25T08:50:16.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:16.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj180 /dev/null 2024-04-25T08:50:16.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:16.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj181 /dev/null 2024-04-25T08:50:17.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:17.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj182 /dev/null 2024-04-25T08:50:17.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:17.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj183 /dev/null 2024-04-25T08:50:17.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:17.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj184 /dev/null 2024-04-25T08:50:17.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:17.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj185 /dev/null 2024-04-25T08:50:17.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:17.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj186 /dev/null 2024-04-25T08:50:17.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:17.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj187 /dev/null 2024-04-25T08:50:17.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:17.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj188 /dev/null 2024-04-25T08:50:17.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:17.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj189 /dev/null 2024-04-25T08:50:17.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:17.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj190 /dev/null 2024-04-25T08:50:18.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:18.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj191 /dev/null 2024-04-25T08:50:18.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:18.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj192 /dev/null 2024-04-25T08:50:18.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:18.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj193 /dev/null 2024-04-25T08:50:18.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:18.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj194 /dev/null 2024-04-25T08:50:18.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:18.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj195 /dev/null 2024-04-25T08:50:18.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:18.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj196 /dev/null 2024-04-25T08:50:18.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:18.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj197 /dev/null 2024-04-25T08:50:19.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:19.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj198 /dev/null 2024-04-25T08:50:19.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:19.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj199 /dev/null 2024-04-25T08:50:19.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:478: TEST_recovery_multi: for i in $(seq $(expr $half + 1) $objects) 2024-04-25T08:50:19.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:480: TEST_recovery_multi: rados -p test put obj200 /dev/null 2024-04-25T08:50:19.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: get_pg test obj1 2024-04-25T08:50:19.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:50:19.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:50:19.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:50:19.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:50:19.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:483: TEST_recovery_multi: local PG=1.0 2024-04-25T08:50:19.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: get_not_primary test obj200 2024-04-25T08:50:19.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:50:19.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj200 2024-04-25T08:50:19.924 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj200 2024-04-25T08:50:19.924 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:50:19.924 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj200 2024-04-25T08:50:19.924 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj200 2024-04-25T08:50:19.924 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:50:20.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=0 2024-04-25T08:50:20.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj200 2024-04-25T08:50:20.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 0)) | .[0]' 2024-04-25T08:50:20.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:484: TEST_recovery_multi: local otherosd=2 2024-04-25T08:50:20.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:486: TEST_recovery_multi: ceph osd unset noout 2024-04-25T08:50:21.574 INFO:tasks.workunit.client.0.smithi134.stderr:noout is unset 2024-04-25T08:50:21.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:487: TEST_recovery_multi: ceph osd out osd.1 osd.2 2024-04-25T08:50:22.247 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already out. osd.2 is already out. 2024-04-25T08:50:22.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:488: TEST_recovery_multi: activate_osd td/osd-recovery-stats 1 2024-04-25T08:50:22.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-recovery-stats 2024-04-25T08:50:22.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T08:50:22.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T08:50:22.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T08:50:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:50:22.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T08:50:22.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:50:22.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:50:22.268 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:50:22.268 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:50:22.268 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:50:22.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:50:22.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:50:22.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:50:22.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:50:22.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:50:22.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:50:22.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:50:22.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:50:22.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:50:22.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T08:50:22.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T08:50:22.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:50:22.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T08:50:22.271 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:50:22.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:50:22.272 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-recovery-stats/1/whoami 2024-04-25T08:50:22.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T08:50:22.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T08:50:22.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T08:50:22.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T08:50:22.294 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:50:22.301+0000 7f6a2af377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:22.310 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:50:22.321+0000 7f6a2af377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:22.312 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:50:22.321+0000 7f6a2af377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:50:22.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T08:50:22.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:50:22.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:50:22.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:50:22.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:50:22.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:22.731 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:50:22.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:50:22.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:22.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:50:22.880 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:50:22.889+0000 7f6a2af377c0 -1 Falling back to public interface 2024-04-25T08:50:23.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:24.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:24.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:24.177 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:50:24.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:50:24.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:24.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:50:24.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:25.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:25.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:25.806 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:50:25.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:50:25.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:25.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:50:26.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:50:26.318 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:50:26.325+0000 7f6a2af377c0 -1 osd.1 45 log_to_monitors true 2024-04-25T08:50:27.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:50:27.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:50:27.273 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:50:27.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:50:27.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:50:27.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:50:27.698 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up out weight 0 up_from 58 up_thru 44 down_at 46 last_clean_interval [10,45) [v2:127.0.0.1:6810/2458703929,v1:127.0.0.1:6811/2458703929] [v2:127.0.0.1:6812/2458703929,v1:127.0.0.1:6813/2458703929] exists,up 1c639d52-e9b6-4a3a-9c35-0fd9e77a7ddb 2024-04-25T08:50:27.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:50:27.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:50:27.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:50:27.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:489: TEST_recovery_multi: sleep 3 2024-04-25T08:50:30.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:491: TEST_recovery_multi: ceph osd pool set test size 4 2024-04-25T08:50:31.580 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 4 2024-04-25T08:50:31.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:492: TEST_recovery_multi: ceph osd unset norecover 2024-04-25T08:50:32.907 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is unset 2024-04-25T08:50:32.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:50:32.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:50:32.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:50:32.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:50:32.931 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:50:33.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:493: TEST_recovery_multi: ceph tell osd.0 debug kick_recovery_wq 0 2024-04-25T08:50:33.630 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:50:33.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:494: TEST_recovery_multi: sleep 2 2024-04-25T08:50:35.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:496: TEST_recovery_multi: wait_for_clean 2024-04-25T08:50:35.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:50:35.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:50:35.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:50:35.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:50:35.649 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:50:35.649 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:50:35.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:50:35.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:50:35.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:50:35.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:50:35.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:50:35.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:50:35.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:50:35.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:50:35.773 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:50:36.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:50:36.329 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:50:36.329 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:50:36.329 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:50:36.329 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T08:50:36.329 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T08:50:36.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:50:36.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:50:36.330 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:50:36.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=210453397513 2024-04-25T08:50:36.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 210453397513 2024-04-25T08:50:36.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513' 2024-04-25T08:50:36.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:50:36.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:50:36.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=249108103172 2024-04-25T08:50:36.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 249108103172 2024-04-25T08:50:36.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103172' 2024-04-25T08:50:36.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:50:36.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:50:36.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509491 2024-04-25T08:50:36.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509491 2024-04-25T08:50:36.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103172 2-64424509491' 2024-04-25T08:50:36.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:50:36.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:50:37.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345957 2024-04-25T08:50:37.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345957 2024-04-25T08:50:37.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103172 2-64424509491 3-85899345957' 2024-04-25T08:50:37.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:50:37.071 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:50:37.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182428 2024-04-25T08:50:37.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182428 2024-04-25T08:50:37.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103172 2-64424509491 3-85899345957 4-107374182428' 2024-04-25T08:50:37.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:50:37.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:50:37.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018906 2024-04-25T08:50:37.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018906 2024-04-25T08:50:37.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-210453397513 1-249108103172 2-64424509491 3-85899345957 4-107374182428 5-128849018906' 2024-04-25T08:50:37.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:50:37.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-210453397513 2024-04-25T08:50:37.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:50:37.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:50:37.381 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-210453397513 2024-04-25T08:50:37.381 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:50:37.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=210453397513 2024-04-25T08:50:37.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 210453397513' 2024-04-25T08:50:37.382 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 210453397513 2024-04-25T08:50:37.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:50:37.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 210453397511 -lt 210453397513 2024-04-25T08:50:37.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:50:38.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:50:38.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:50:39.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 210453397514 -lt 210453397513 2024-04-25T08:50:39.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:50:39.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-249108103172 2024-04-25T08:50:39.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:50:39.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:50:39.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-249108103172 2024-04-25T08:50:39.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:50:39.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=249108103172 2024-04-25T08:50:39.232 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 249108103172 2024-04-25T08:50:39.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 249108103172' 2024-04-25T08:50:39.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:50:39.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 249108103173 -lt 249108103172 2024-04-25T08:50:39.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:50:39.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509491 2024-04-25T08:50:39.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:50:39.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:50:39.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509491 2024-04-25T08:50:39.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:50:39.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509491 2024-04-25T08:50:39.871 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509491 2024-04-25T08:50:39.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509491' 2024-04-25T08:50:39.872 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:50:40.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509491 -lt 64424509491 2024-04-25T08:50:40.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:50:40.561 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345957 2024-04-25T08:50:40.561 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:50:40.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:50:40.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345957 2024-04-25T08:50:40.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:50:40.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345957 2024-04-25T08:50:40.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345957' 2024-04-25T08:50:40.565 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345957 2024-04-25T08:50:40.565 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:50:41.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345958 -lt 85899345957 2024-04-25T08:50:41.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:50:41.246 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182428 2024-04-25T08:50:41.246 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:50:41.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:50:41.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182428 2024-04-25T08:50:41.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:50:41.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182428 2024-04-25T08:50:41.249 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182428 2024-04-25T08:50:41.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182428' 2024-04-25T08:50:41.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:50:41.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182429 -lt 107374182428 2024-04-25T08:50:41.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:50:41.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018906 2024-04-25T08:50:41.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:50:41.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:50:41.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018906 2024-04-25T08:50:41.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:50:41.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018906 2024-04-25T08:50:41.825 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018906 2024-04-25T08:50:41.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018906' 2024-04-25T08:50:41.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:50:42.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018907 -lt 128849018906 2024-04-25T08:50:42.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:50:42.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:42.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:43.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:50:43.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:43.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:43.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:43.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:43.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:43.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:43.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:43.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:50:43.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:43.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:43.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:50:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:50:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:50:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:50:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:44.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:50:44.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:44.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:44.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:44.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:44.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:44.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:44.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:44.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:44.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:50:44.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:44.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:44.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:45.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:50:45.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:50:45.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:45.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:45.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:45.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:45.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:45.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:45.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:46.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:50:46.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:50:46.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:50:46.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:46.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:50:46.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:46.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:46.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:46.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:46.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:46.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:46.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:46.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:46.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:50:46.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:46.920 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:46.920 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:47.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:50:47.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:50:47.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:47.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:47.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:47.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:47.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:47.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:47.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:48.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:50:48.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:50:48.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:50:48.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:48.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:50:48.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:48.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:48.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:48.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:48.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:48.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:48.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:48.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:48.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:50:48.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:48.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:48.859 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:49.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:50:49.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:50:49.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:49.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:49.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:49.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:49.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:49.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:49.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:50.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:50:50.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:50:50.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:50:50.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:50.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:50:50.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:50.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:50.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:50.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:50.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:50.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:50.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:50.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:50.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:50:50.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:50.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:50.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:51.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:50:51.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:50:51.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:51.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:51.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:51.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:51.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:51.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:51.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:51.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:50:51.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:50:51.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:50:51.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:51.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:50:52.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:52.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:52.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:52.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:52.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:52.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:52.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:52.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:52.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:50:52.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:52.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:52.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:53.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:50:53.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:50:53.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:53.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:53.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:53.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:53.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:53.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:53.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:53.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:50:53.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:50:53.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:50:53.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:53.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:50:54.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:54.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:54.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:54.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:54.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:54.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:54.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:54.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:54.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:50:54.615 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:54.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:54.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:55.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:50:55.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:50:55.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:55.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:55.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:55.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:55.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:55.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:55.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:55.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:50:55.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:50:55.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:50:55.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:55.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:50:56.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:56.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:56.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:56.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:56.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:56.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:56.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:56.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:56.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:50:56.412 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:56.412 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:56.413 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:57.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:50:57.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:50:57.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:57.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:57.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:57.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:57.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:57.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:57.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:57.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:50:57.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:50:57.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:50:57.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:57.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:50:57.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:57.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:57.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:57.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:50:58.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:50:58.190 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:50:58.190 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:50:58.190 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:50:58.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:50:58.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:50:58.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:50:58.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:50:58.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:50:58.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:50:58.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:50:58.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:50:58.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:50:59.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:50:59.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:50:59.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:50:59.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:50:59.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:50:59.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:50:59.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:50:59.719 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:50:59.719 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:50:59.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:50:59.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:50:59.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:50:59.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:51:00.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:51:00.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:51:00.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:51:00.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:51:01.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:51:01.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:51:01.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:51:01.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:51:01.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:51:01.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:51:01.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:51:01.050 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:51:01.050 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:51:01.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=9 2024-04-25T08:51:01.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 9 '!=' null 2024-04-25T08:51:01.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:51:01.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:51:01.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:51:01.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:51:01.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:51:01.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:51:01.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:51:01.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:51:01.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:51:01.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:51:01.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:51:02.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:51:02.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:51:02.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:51:02.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:51:02.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:51:02.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:51:02.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:51:02.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:51:02.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:51:02.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:51:02.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:51:02.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:51:02.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:51:03.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T08:51:03.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T08:51:03.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:51:03.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:51:03.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:51:03.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:51:03.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:51:03.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:51:03.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:51:03.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:51:03.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:51:03.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:51:03.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:51:03.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:51:03.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:51:03.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:51:03.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:51:04.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:51:04.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:51:04.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:51:04.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:51:04.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:51:04.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:51:04.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:51:04.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:51:04.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:51:04.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T08:51:04.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T08:51:04.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:51:04.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:51:04.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:51:05.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:51:05.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:51:05.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:51:05.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:51:05.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:51:05.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:51:05.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:51:05.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:51:05.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:51:05.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:51:05.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:51:05.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:51:06.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:51:06.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:51:06.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:51:06.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: get_primary test obj1 2024-04-25T08:51:06.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:51:06.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:51:06.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:51:06.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:51:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:499: TEST_recovery_multi: primary=0 2024-04-25T08:51:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:501: TEST_recovery_multi: local log=td/osd-recovery-stats/osd.0.log 2024-04-25T08:51:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:502: TEST_recovery_multi: check td/osd-recovery-stats 1.0 0 replicated 399 0 300 0 99 0 2024-04-25T08:51:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:51:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:51:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=0 2024-04-25T08:51:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=399 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=300 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=99 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.0.log 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:51:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:51:06.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[0,4,3,5]/[0,3]' 2024-04-25T08:51:06.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:51:06.462 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:51:06.463 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:51:06.463 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:51:06.463 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:51:06.463 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:51:06.463 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:51:06.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=399 2024-04-25T08:51:06.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 399 399 2024-04-25T08:51:06.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=399 2024-04-25T08:51:06.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:51:06.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=399 2024-04-25T08:51:06.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:51:06.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:51:06.469 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:51:06.469 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:51:06.469 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:51:06.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:51:06.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:51:06.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:51:06.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:51:06.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:51:06.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:51:06.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:51:06.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:51:06.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:51:06.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:51:06.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:51:06.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:51:06.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:51:06.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=300 2024-04-25T08:51:06.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 300 300 2024-04-25T08:51:06.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=300 2024-04-25T08:51:06.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:51:06.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=300 2024-04-25T08:51:06.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:51:06.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:51:06.487 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:51:06.487 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:51:06.487 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:51:06.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:51:06.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:51:06.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:51:06.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:51:06.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:51:06.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:51:06.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 99 ']' 2024-04-25T08:51:06.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 0' 2024-04-25T08:51:06.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:51:06.502 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:51:06.502 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [0,4,3,5]/[0,3] ' 2024-04-25T08:51:06.502 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-04-25T08:51:06.502 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-04-25T08:51:06.502 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:51:06.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=99 2024-04-25T08:51:06.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 99 99 2024-04-25T08:51:06.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=99 2024-04-25T08:51:06.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:51:06.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=99 2024-04-25T08:51:06.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:51:06.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:51:06.507 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 0 ' td/osd-recovery-stats/osd.0.log 2024-04-25T08:51:06.507 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-04-25T08:51:06.507 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:51:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-04-25T08:51:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-04-25T08:51:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:51:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:51:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:51:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:51:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:504: TEST_recovery_multi: delete_pool test 2024-04-25T08:51:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:51:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:51:07.099 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T08:51:07.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:505: TEST_recovery_multi: kill_daemons td/osd-recovery-stats 2024-04-25T08:51:07.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:51:07.119 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:51:07.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:51:07.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:51:07.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:51:12.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:51:12.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:51:12.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:51:12.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:51:12.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:51:12.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:51:12.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:51:12.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:51:12.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:51:12.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:51:12.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:51:12.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:51:12.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:51:12.450 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:51:12.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:51:12.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:51:12.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:51:12.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:51:12.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:51:12.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:51:12.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:51:12.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:51:12.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:51:12.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:51:12.501 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:51:12.501 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:12.501 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:51:12.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T08:51:12.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:51:12.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:51:12.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:51:12.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:51:12.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:51:12.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:51:12.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:51:12.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:51:12.506 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:51:12.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:51:12.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:51:12.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:51:12.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:51:12.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:51:12.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:51:12.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:51:12.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:51:12.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:51:12.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:51:12.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:51:12.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:12.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:51:12.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T08:51:12.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:51:12.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:51:12.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:51:12.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:51:12.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:12.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:51:12.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.280166 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_replicated_out1 td/osd-recovery-stats 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:167: TEST_recovery_replicated_out1: local dir=td/osd-recovery-stats 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:169: TEST_recovery_replicated_out1: do_recovery_out1 td/osd-recovery-stats replicated 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:117: do_recovery_out1: local dir=td/osd-recovery-stats 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:118: do_recovery_out1: shift 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:119: do_recovery_out1: local type=replicated 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:121: do_recovery_out1: run_mon td/osd-recovery-stats a 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:51:12.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:51:12.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:51:12.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:51:12.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:51:12.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:51:12.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:51:12.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:51:12.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:51:12.992 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:51:12.992 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:12.992 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:51:12.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:51:12.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:51:13.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:51:13.220 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:51:13.220 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:51:13.220 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:51:13.220 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:51:13.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:51:13.221 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:51:13.221 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:51:13.221 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:51:13.222 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:51:13.222 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:13.222 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:51:13.222 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:51:13.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:51:13.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get fsid 2024-04-25T08:51:13.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:51:13.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:51:13.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:51:13.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:51:13.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get mon_host 2024-04-25T08:51:13.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:122: do_recovery_out1: run_mgr td/osd-recovery-stats x 2024-04-25T08:51:13.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:51:13.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:51:13.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:51:13.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:51:13.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:51:13.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:51:14.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:51:14.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:51:14.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:51:14.352 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:51:14.352 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:14.352 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:51:14.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:51:14.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:51:14.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:51:14.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:123: do_recovery_out1: run_osd td/osd-recovery-stats 0 2024-04-25T08:51:14.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:51:14.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:51:14.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:51:14.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:51:14.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:14.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:51:14.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:51:14.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:51:14.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3e05e3a2-8685-4a8c-b78c-56272cd86465 2024-04-25T08:51:14.389 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 3e05e3a2-8685-4a8c-b78c-56272cd86465 2024-04-25T08:51:14.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3e05e3a2-8685-4a8c-b78c-56272cd86465' 2024-04-25T08:51:14.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:51:14.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCCGSpmWU6hGBAAe//EoMBrrxCHINQVS4Y1GA== 2024-04-25T08:51:14.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCCGSpmWU6hGBAAe//EoMBrrxCHINQVS4Y1GA=="}' 2024-04-25T08:51:14.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3e05e3a2-8685-4a8c-b78c-56272cd86465 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:51:14.739 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:51:14.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:51:14.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCCGSpmWU6hGBAAe//EoMBrrxCHINQVS4Y1GA== --osd-uuid 3e05e3a2-8685-4a8c-b78c-56272cd86465 2024-04-25T08:51:14.789 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:14.796+0000 7f2362f887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:14.815 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:14.824+0000 7f2362f887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:14.817 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:14.824+0000 7f2362f887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:14.818 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:14.824+0000 7f2362f887c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:51:14.818 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:14.824+0000 7f2362f887c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:51:14.818 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:14.824+0000 7f2362f887c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:51:14.856 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:14.864+0000 7f2362f887c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:51:21.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:51:21.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:51:21.225 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:51:21.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:51:21.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:51:21.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:51:21.779 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:51:21.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:51:21.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:51:21.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:51:21.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:51:21.810 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:21.820+0000 7f2b3ede67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:21.846 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:21.856+0000 7f2b3ede67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:21.848 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:21.856+0000 7f2b3ede67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:22.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:51:22.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:51:22.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:51:22.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:51:22.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:51:22.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:22.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:51:22.220 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:51:22.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:22.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:22.408 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:22.415+0000 7f2b3ede67c0 -1 Falling back to public interface 2024-04-25T08:51:22.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:23.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:23.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:23.636 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:51:23.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:51:23.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:23.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:25.037 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:25.043+0000 7f2b3ede67c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:51:25.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:25.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:25.051 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:51:25.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:51:25.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:25.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:25.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:26.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:26.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:26.683 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:51:26.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:51:26.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:26.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:27.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:28.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:28.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:28.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:51:28.172 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:51:28.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:28.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:28.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:29.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:29.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:29.633 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:51:29.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:51:29.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:29.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:30.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:31.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:31.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:31.132 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:51:31.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:51:31.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:31.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:31.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:32.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:32.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:32.600 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:51:32.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:51:32.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:32.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:33.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:34.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:34.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:34.116 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:51:34.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:51:34.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:34.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:34.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:35.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:35.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:35.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:51:35.532 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:51:35.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:35.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:35.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:36.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:36.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:36.984 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:51:36.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:51:36.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:36.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:37.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:38.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:38.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:38.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:51:38.424 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:51:38.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:38.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:39.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:40.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:40.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:40.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:51:40.478 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:51:40.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:40.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:41.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:42.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:42.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:42.441 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:51:42.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:51:42.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:42.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:43.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:44.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:44.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:44.115 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:51:44.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:51:44.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:44.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:44.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:45.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:45.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:45.846 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:51:45.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:51:45.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:45.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:46.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:47.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:47.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:47.977 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:51:47.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:51:47.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:47.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:48.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:51:49.727 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:51:49.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:49.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:51:50.180 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2730465225,v1:127.0.0.1:6803/2730465225] [v2:127.0.0.1:6804/2730465225,v1:127.0.0.1:6805/2730465225] exists,up 3e05e3a2-8685-4a8c-b78c-56272cd86465 2024-04-25T08:51:50.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:51:50.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:51:50.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:51:50.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:124: do_recovery_out1: run_osd td/osd-recovery-stats 1 2024-04-25T08:51:50.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:51:50.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:51:50.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:51:50.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:51:50.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:51:50.185 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:51:50.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:51:50.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:51:50.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:51:50.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:51:50.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:51:50.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:51:50.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0f5251db-1a5d-43f4-b2a0-1432fb9c9b53 2024-04-25T08:51:50.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0f5251db-1a5d-43f4-b2a0-1432fb9c9b53' 2024-04-25T08:51:50.188 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 0f5251db-1a5d-43f4-b2a0-1432fb9c9b53 2024-04-25T08:51:50.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:51:50.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmGSpmbhoDDRAAlnSq2yyVJPoAlmTckBu5rw== 2024-04-25T08:51:50.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmGSpmbhoDDRAAlnSq2yyVJPoAlmTckBu5rw=="}' 2024-04-25T08:51:50.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0f5251db-1a5d-43f4-b2a0-1432fb9c9b53 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:51:50.764 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:51:50.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:51:50.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmGSpmbhoDDRAAlnSq2yyVJPoAlmTckBu5rw== --osd-uuid 0f5251db-1a5d-43f4-b2a0-1432fb9c9b53 2024-04-25T08:51:50.815 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:50.823+0000 7f5f341087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:50.833 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:50.839+0000 7f5f341087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:50.834 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:50.843+0000 7f5f341087c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:50.835 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:50.843+0000 7f5f341087c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:51:50.835 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:50.843+0000 7f5f341087c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:51:50.835 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:50.843+0000 7f5f341087c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:51:50.873 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:50.879+0000 7f5f341087c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:51:57.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:51:57.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:51:57.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:51:57.562 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:51:57.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:51:58.115 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:51:58.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:51:58.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:51:58.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:51:58.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:51:58.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:51:58.140 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:58.147+0000 7f49f5d0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:58.142 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:58.151+0000 7f49f5d0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:58.144 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:58.151+0000 7f49f5d0b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:51:58.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:51:58.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:51:58.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:51:58.522 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:51:58.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:51:58.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:51:58.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:58.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:51:58.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:58.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:51:58.695 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:51:58.703+0000 7f49f5d0b7c0 -1 Falling back to public interface 2024-04-25T08:51:58.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:51:59.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:51:59.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:51:59.931 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:51:59.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:51:59.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:51:59.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:00.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:01.118 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:01.122+0000 7f49f5d0b7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:52:01.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:01.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:01.347 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:52:01.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:52:01.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:01.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:01.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:02.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:02.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:02.896 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:52:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:52:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:03.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:04.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:04.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:04.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:52:04.482 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:52:04.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:04.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:05.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:06.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:06.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:06.007 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:52:06.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:52:06.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:06.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:07.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:07.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:07.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:52:07.432 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:52:07.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:07.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:07.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:08.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:08.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:08.889 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:52:08.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:52:08.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:08.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:09.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:10.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:10.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:10.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:52:10.399 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:52:10.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:10.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:11.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:12.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:12.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:12.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:52:12.007 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:52:12.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:12.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:12.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:13.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:13.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:13.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:52:13.430 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:52:13.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:13.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:13.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:14.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:14.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:14.900 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:52:14.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:52:14.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:14.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:15.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:16.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:16.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:16.522 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:52:16.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:52:16.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:16.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:17.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:18.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:18.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:18.750 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:52:18.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:52:18.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:18.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:19.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:20.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:20.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:20.567 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:52:20.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:52:20.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:20.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:21.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:22.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:22.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:22.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:52:22.383 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:52:22.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:22.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:23.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:24.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:24.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:24.131 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:52:24.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:52:24.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:24.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:25.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:26.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:26.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:26.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:52:26.480 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:52:26.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:26.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:52:26.995 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1639025095,v1:127.0.0.1:6811/1639025095] [v2:127.0.0.1:6812/1639025095,v1:127.0.0.1:6813/1639025095] exists,up 0f5251db-1a5d-43f4-b2a0-1432fb9c9b53 2024-04-25T08:52:26.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:52:26.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:52:26.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:52:26.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:125: do_recovery_out1: run_osd td/osd-recovery-stats 2 2024-04-25T08:52:26.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:52:26.999 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:52:27.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:52:27.000 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:52:27.000 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:52:27.000 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:52:27.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:52:27.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:52:27.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:52:27.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:52:27.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:52:27.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b9875184-b9d4-4325-ae2c-7394b48f477f 2024-04-25T08:52:27.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b9875184-b9d4-4325-ae2c-7394b48f477f' 2024-04-25T08:52:27.003 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 b9875184-b9d4-4325-ae2c-7394b48f477f 2024-04-25T08:52:27.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:52:27.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLGSpmIzcfAhAAOwnzjdHSrSN4nHTEIpO5Wg== 2024-04-25T08:52:27.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLGSpmIzcfAhAAOwnzjdHSrSN4nHTEIpO5Wg=="}' 2024-04-25T08:52:27.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b9875184-b9d4-4325-ae2c-7394b48f477f -i td/osd-recovery-stats/2/new.json 2024-04-25T08:52:27.543 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:52:27.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:52:27.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLGSpmIzcfAhAAOwnzjdHSrSN4nHTEIpO5Wg== --osd-uuid b9875184-b9d4-4325-ae2c-7394b48f477f 2024-04-25T08:52:27.594 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:27.598+0000 7f40848f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:27.606 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:27.610+0000 7f40848f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:27.607 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:27.614+0000 7f40848f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:27.607 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:27.614+0000 7f40848f47c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:52:27.607 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:27.614+0000 7f40848f47c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:52:27.607 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:27.614+0000 7f40848f47c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:52:27.645 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:27.650+0000 7f40848f47c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:52:34.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:52:34.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:52:34.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:52:34.434 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T08:52:34.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:52:35.012 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T08:52:35.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:52:35.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:52:35.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:52:35.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:52:35.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:52:35.044 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:35.054+0000 7f03624e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:35.063 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:35.070+0000 7f03624e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:35.064 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:35.070+0000 7f03624e77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:52:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:52:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:52:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:52:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:52:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:52:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:35.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:52:35.445 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:52:35.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:35.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:35.623 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:35.634+0000 7f03624e77c0 -1 Falling back to public interface 2024-04-25T08:52:35.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:36.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:36.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:36.878 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:52:36.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:52:36.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:36.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:37.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:38.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:38.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:38.285 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:52:38.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:52:38.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:38.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:38.346 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:52:38.358+0000 7f03624e77c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:52:38.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:39.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:39.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:39.746 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:52:39.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:52:39.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:39.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:40.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:41.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:41.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:41.363 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:52:41.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:52:41.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:41.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:41.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:42.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:42.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:42.849 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:52:42.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:52:42.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:42.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:43.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:44.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:44.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:44.363 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:52:44.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:52:44.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:44.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:44.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:45.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:45.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:45.845 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:52:45.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:52:45.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:45.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:46.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:47.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:47.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:47.296 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:52:47.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:52:47.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:47.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:47.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:48.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:48.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:48.762 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:52:48.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:52:48.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:48.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:49.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:50.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:50.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:50.415 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:52:50.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:52:50.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:50.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:50.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:51.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:51.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:51.878 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:52:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:52:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:52.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:53.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:53.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:53.384 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:52:53.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:52:53.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:53.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:54.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:55.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:55.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:55.413 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:52:55.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:52:55.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:55.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:56.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:57.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:57.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:57.720 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:52:57.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:52:57.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:57.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:52:58.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:52:59.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:52:59.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:52:59.297 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:52:59.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:52:59.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:52:59.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:00.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:01.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:53:01.183 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:53:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:01.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:02.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:02.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:02.736 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:53:02.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:53:02.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:02.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:03.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:04.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:04.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:04.512 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:53:04.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:53:04.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:04.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:04.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:05.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:05.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:05.950 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T08:53:05.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T08:53:05.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:05.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:53:06.489 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1941747300,v1:127.0.0.1:6819/1941747300] [v2:127.0.0.1:6820/1941747300,v1:127.0.0.1:6821/1941747300] exists,up b9875184-b9d4-4325-ae2c-7394b48f477f 2024-04-25T08:53:06.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:53:06.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:53:06.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:126: do_recovery_out1: run_osd td/osd-recovery-stats 3 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:53:06.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:53:06.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:53:06.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:53:06.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:53:06.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:53:06.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:53:06.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:53:06.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:53:06.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:53:06.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d326ef70-6430-41c8-a9bb-1307013abc52 2024-04-25T08:53:06.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 d326ef70-6430-41c8-a9bb-1307013abc52' 2024-04-25T08:53:06.497 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 d326ef70-6430-41c8-a9bb-1307013abc52 2024-04-25T08:53:06.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:53:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDyGSpmQrd2HxAAUNeXA6yRRPjPbPw5s5LE/Q== 2024-04-25T08:53:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDyGSpmQrd2HxAAUNeXA6yRRPjPbPw5s5LE/Q=="}' 2024-04-25T08:53:06.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d326ef70-6430-41c8-a9bb-1307013abc52 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:53:07.022 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:53:07.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:53:07.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDyGSpmQrd2HxAAUNeXA6yRRPjPbPw5s5LE/Q== --osd-uuid d326ef70-6430-41c8-a9bb-1307013abc52 2024-04-25T08:53:07.065 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:07.073+0000 7f1268b3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:07.140 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:07.149+0000 7f1268b3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:07.142 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:07.153+0000 7f1268b3b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:07.142 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:07.153+0000 7f1268b3b7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:53:07.143 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:07.153+0000 7f1268b3b7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:53:07.143 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:07.153+0000 7f1268b3b7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:53:07.239 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:07.249+0000 7f1268b3b7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:53:14.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:53:14.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:53:14.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:53:14.333 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T08:53:14.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:53:14.901 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T08:53:14.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:53:14.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:53:14.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:53:14.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:53:14.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:53:14.927 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:14.937+0000 7f255d62c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:14.929 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:14.937+0000 7f255d62c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:14.930 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:14.941+0000 7f255d62c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:15.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:53:15.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:53:15.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:53:15.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:53:15.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:53:15.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:15.350 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:53:15.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:53:15.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:15.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:15.507 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:15.517+0000 7f255d62c7c0 -1 Falling back to public interface 2024-04-25T08:53:15.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:16.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:16.790 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:53:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:53:16.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:16.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:17.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:18.140 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:18.149+0000 7f255d62c7c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:53:18.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:18.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:18.249 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:53:18.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:53:18.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:18.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:18.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:19.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:19.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:19.806 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:53:19.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:53:19.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:19.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:20.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:21.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:21.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:21.344 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:53:21.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:53:21.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:21.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:21.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:22.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:22.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:22.914 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:53:22.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:53:22.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:22.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:23.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:24.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:24.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:24.533 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:53:24.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:53:24.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:24.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:24.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:25.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:25.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:25.954 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:53:25.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:53:25.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:25.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:26.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:27.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:27.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:27.577 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:53:27.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:53:27.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:27.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:28.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:29.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:29.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:29.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:53:29.176 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:53:29.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:29.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:29.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:30.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:30.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:30.802 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:53:30.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:53:30.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:30.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:31.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:32.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:32.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:32.351 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:53:32.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:53:32.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:32.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:33.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:34.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:34.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:34.008 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:53:34.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:53:34.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:34.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:35.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:36.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:36.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:36.069 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:53:36.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:53:36.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:36.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:36.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:37.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:37.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:37.855 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:53:37.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:53:37.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:37.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:38.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:39.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:39.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:39.596 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:53:39.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:53:39.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:39.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:40.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:41.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:41.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:41.652 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:53:41.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:53:41.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:41.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:42.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:43.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:43.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:43.761 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:53:43.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:53:43.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:43.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:44.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:45.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:45.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:45.411 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:53:45.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:53:45.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:45.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:53:45.835 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4089692762,v1:127.0.0.1:6827/4089692762] [v2:127.0.0.1:6828/4089692762,v1:127.0.0.1:6829/4089692762] exists,up d326ef70-6430-41c8-a9bb-1307013abc52 2024-04-25T08:53:45.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:127: do_recovery_out1: run_osd td/osd-recovery-stats 4 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:53:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:53:45.839 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:53:45.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:53:45.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:53:45.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:53:45.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:53:45.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:53:45.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c0a1ce45-1767-437b-bcb7-f1f2ede20165 2024-04-25T08:53:45.842 INFO:tasks.workunit.client.0.smithi134.stdout:add osd4 c0a1ce45-1767-437b-bcb7-f1f2ede20165 2024-04-25T08:53:45.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 c0a1ce45-1767-437b-bcb7-f1f2ede20165' 2024-04-25T08:53:45.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:53:45.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAZGipmACoJNBAAgNrqacxyDQd2hmhlqtJ9ug== 2024-04-25T08:53:45.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAZGipmACoJNBAAgNrqacxyDQd2hmhlqtJ9ug=="}' 2024-04-25T08:53:45.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c0a1ce45-1767-437b-bcb7-f1f2ede20165 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:53:46.392 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:53:46.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:53:46.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZGipmACoJNBAAgNrqacxyDQd2hmhlqtJ9ug== --osd-uuid c0a1ce45-1767-437b-bcb7-f1f2ede20165 2024-04-25T08:53:46.437 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:46.444+0000 7f3fd78697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:46.439 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:46.448+0000 7f3fd78697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:46.440 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:46.448+0000 7f3fd78697c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:46.441 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:46.448+0000 7f3fd78697c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:53:46.441 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:46.448+0000 7f3fd78697c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:53:46.441 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:46.448+0000 7f3fd78697c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:53:46.513 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:46.520+0000 7f3fd78697c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:53:53.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:53:53.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:53:53.204 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd4 key to auth repository 2024-04-25T08:53:53.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:53:53.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:53:53.764 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.4 2024-04-25T08:53:53.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:53:53.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:53:53.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:53:53.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:53:53.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:53:53.796 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:53.804+0000 7f8e10cdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:53.816 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:53.824+0000 7f8e10cdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:53.817 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:53.824+0000 7f8e10cdd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:53:54.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:53:54.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:53:54.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:53:54.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:53:54.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:53:54.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:54.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:53:54.180 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:53:54.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:54.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:53:54.403 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:54.412+0000 7f8e10cdd7c0 -1 Falling back to public interface 2024-04-25T08:53:54.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:55.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:55.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:55.591 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:53:55.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:53:55.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:55.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:53:55.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:56.963 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:53:56.972+0000 7f8e10cdd7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:53:56.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:56.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:56.995 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:53:56.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:53:56.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:56.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:53:57.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:53:58.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:53:58.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:53:58.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:53:58.444 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:53:58.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:53:58.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:53:59.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:00.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:00.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:00.047 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:54:00.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:54:00.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:00.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:00.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:01.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:01.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:01.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:54:01.547 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:54:01.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:01.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:02.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:03.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:03.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:03.182 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:54:03.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:54:03.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:03.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:03.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:04.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:04.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:04.750 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:54:04.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:54:04.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:04.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:05.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:06.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:06.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:06.393 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:54:06.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:54:06.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:06.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:07.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:08.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:08.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:08.020 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:54:08.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:54:08.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:08.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:08.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:09.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:09.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:09.452 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:54:09.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:54:09.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:09.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:09.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:10.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:10.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:10.881 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:54:10.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:54:10.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:10.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:11.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:12.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:12.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:12.339 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:54:12.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:54:12.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:12.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:13.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:14.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:14.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:14.337 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:54:14.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:54:14.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:14.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:15.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:16.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:16.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:16.288 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:54:16.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:54:16.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:16.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:17.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:18.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:18.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:18.200 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:54:18.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:54:18.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:18.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:18.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:19.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:19.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:19.977 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:54:19.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:54:19.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:19.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:20.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:21.710 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:54:21.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:21.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:21.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:54:21.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:21.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:22.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:23.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:23.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:23.406 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:54:23.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:54:23.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:23.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:54:23.818 INFO:tasks.workunit.client.0.smithi134.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/28679745,v1:127.0.0.1:6835/28679745] [v2:127.0.0.1:6836/28679745,v1:127.0.0.1:6837/28679745] exists,up c0a1ce45-1767-437b-bcb7-f1f2ede20165 2024-04-25T08:54:23.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:128: do_recovery_out1: run_osd td/osd-recovery-stats 5 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:54:23.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:54:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:54:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:54:23.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:54:23.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ed23a812-c4c0-4c04-bd46-fc8a5ce27f88 2024-04-25T08:54:23.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 ed23a812-c4c0-4c04-bd46-fc8a5ce27f88' 2024-04-25T08:54:23.825 INFO:tasks.workunit.client.0.smithi134.stdout:add osd5 ed23a812-c4c0-4c04-bd46-fc8a5ce27f88 2024-04-25T08:54:23.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:54:23.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/GipmCroBMxAAue3aOHY4eHr/ixtnWwzt7g== 2024-04-25T08:54:23.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/GipmCroBMxAAue3aOHY4eHr/ixtnWwzt7g=="}' 2024-04-25T08:54:23.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ed23a812-c4c0-4c04-bd46-fc8a5ce27f88 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:54:24.466 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:54:24.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:54:24.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/GipmCroBMxAAue3aOHY4eHr/ixtnWwzt7g== --osd-uuid ed23a812-c4c0-4c04-bd46-fc8a5ce27f88 2024-04-25T08:54:24.523 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:24.531+0000 7fe27313c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:24.590 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:24.599+0000 7fe27313c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:24.591 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:24.599+0000 7fe27313c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:24.592 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:24.599+0000 7fe27313c7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:54:24.592 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:24.599+0000 7fe27313c7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:54:24.592 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:24.599+0000 7fe27313c7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:54:24.673 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:24.679+0000 7fe27313c7c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:54:31.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:54:31.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:54:31.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:54:31.606 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd5 key to auth repository 2024-04-25T08:54:31.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:54:32.181 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.5 2024-04-25T08:54:32.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:54:32.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:54:32.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:54:32.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:54:32.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:54:32.213 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:32.219+0000 7f0b0cdfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:32.215 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:32.223+0000 7f0b0cdfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:32.216 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:32.223+0000 7f0b0cdfe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:54:32.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:54:32.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:54:32.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:54:32.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:54:32.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:54:32.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:32.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:54:32.572 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:54:32.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:32.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:32.787 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:32.795+0000 7f0b0cdfe7c0 -1 Falling back to public interface 2024-04-25T08:54:33.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:34.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:34.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:34.068 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:54:34.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:54:34.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:34.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:34.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:35.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:35.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:35.476 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:54:35.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:54:35.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:35.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:35.600 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:54:35.607+0000 7f0b0cdfe7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:54:35.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:36.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:36.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:36.975 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:54:36.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:54:36.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:36.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:37.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:38.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:38.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:38.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:54:38.439 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:54:38.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:38.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:38.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:39.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:39.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:39.989 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:54:39.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:54:39.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:39.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:40.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:41.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:41.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:41.442 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:54:41.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:54:41.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:41.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:42.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:43.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:43.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:43.004 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:54:43.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:54:43.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:43.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:43.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:44.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:44.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:44.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:54:44.436 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:54:44.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:44.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:44.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:45.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:45.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:45.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:54:45.868 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:54:45.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:45.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:46.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:47.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:47.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:47.325 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:54:47.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:54:47.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:47.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:47.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:48.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:48.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:48.789 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:54:48.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:54:48.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:48.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:49.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:50.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:50.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:50.207 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:54:50.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:54:50.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:50.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:50.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:51.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:51.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:51.634 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:54:51.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:54:51.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:51.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:52.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:53.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:53.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:53.673 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:54:53.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:54:53.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:53.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:54.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:55.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:55.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:55.396 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:54:55.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:54:55.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:55.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:56.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:57.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:57.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:57.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:54:57.635 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:54:57.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:57.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:54:58.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:54:59.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:54:59.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:54:59.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:54:59.394 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:54:59.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:54:59.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:55:00.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:55:01.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:55:01.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:55:01.219 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:55:01.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:55:01.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:55:01.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:55:01.678 INFO:tasks.workunit.client.0.smithi134.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2090067651,v1:127.0.0.1:6843/2090067651] [v2:127.0.0.1:6844/2090067651,v1:127.0.0.1:6845/2090067651] exists,up ed23a812-c4c0-4c04-bd46-fc8a5ce27f88 2024-04-25T08:55:01.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:55:01.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:55:01.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:55:01.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:130: do_recovery_out1: '[' replicated = erasure ']' 2024-04-25T08:55:01.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:135: do_recovery_out1: create_pool test 1 1 replicated 2024-04-25T08:55:01.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 replicated 2024-04-25T08:55:02.463 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T08:55:02.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:55:03.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:138: do_recovery_out1: wait_for_clean 2024-04-25T08:55:03.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:55:03.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:55:03.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:55:03.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:55:03.483 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:55:03.483 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:55:03.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:55:03.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:55:03.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:55:03.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:55:03.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:55:03.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:55:03.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:55:03.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:55:03.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:55:04.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:55:04.168 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:55:04.168 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:55:04.168 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:55:04.168 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T08:55:04.168 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T08:55:04.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:55:04.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:04.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:55:04.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836527 2024-04-25T08:55:04.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836527 2024-04-25T08:55:04.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527' 2024-04-25T08:55:04.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:04.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:55:04.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672999 2024-04-25T08:55:04.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672999 2024-04-25T08:55:04.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672999' 2024-04-25T08:55:04.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:04.478 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:55:04.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509469 2024-04-25T08:55:04.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509469 2024-04-25T08:55:04.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672999 2-64424509469' 2024-04-25T08:55:04.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:04.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:55:04.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345941 2024-04-25T08:55:04.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345941 2024-04-25T08:55:04.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672999 2-64424509469 3-85899345941' 2024-04-25T08:55:04.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:04.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:55:04.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182415 2024-04-25T08:55:04.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182415 2024-04-25T08:55:04.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672999 2-64424509469 3-85899345941 4-107374182415' 2024-04-25T08:55:04.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:04.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:55:05.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018890 2024-04-25T08:55:05.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018890 2024-04-25T08:55:05.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672999 2-64424509469 3-85899345941 4-107374182415 5-128849018890' 2024-04-25T08:55:05.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:05.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836527 2024-04-25T08:55:05.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:05.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:55:05.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836527 2024-04-25T08:55:05.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:05.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836527 2024-04-25T08:55:05.089 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836527 2024-04-25T08:55:05.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-04-25T08:55:05.089 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:55:05.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836527 -lt 21474836527 2024-04-25T08:55:05.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:05.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672999 2024-04-25T08:55:05.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:05.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:55:05.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672999 2024-04-25T08:55:05.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:05.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672999 2024-04-25T08:55:05.494 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672999 2024-04-25T08:55:05.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672999' 2024-04-25T08:55:05.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:55:05.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672999 -lt 42949672999 2024-04-25T08:55:05.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:05.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509469 2024-04-25T08:55:05.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:05.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:55:05.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:05.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509469 2024-04-25T08:55:05.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509469 2024-04-25T08:55:05.922 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509469 2024-04-25T08:55:05.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509469' 2024-04-25T08:55:05.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:55:06.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509469 -lt 64424509469 2024-04-25T08:55:06.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:06.413 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345941 2024-04-25T08:55:06.413 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:06.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:55:06.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345941 2024-04-25T08:55:06.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:06.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345941 2024-04-25T08:55:06.416 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345941 2024-04-25T08:55:06.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2024-04-25T08:55:06.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:55:06.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345942 -lt 85899345941 2024-04-25T08:55:06.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:06.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182415 2024-04-25T08:55:06.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:06.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:55:06.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182415 2024-04-25T08:55:06.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:06.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182415 2024-04-25T08:55:06.836 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182415 2024-04-25T08:55:06.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-04-25T08:55:06.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:55:07.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182415 -lt 107374182415 2024-04-25T08:55:07.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:07.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018890 2024-04-25T08:55:07.239 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:55:07.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018890 2024-04-25T08:55:07.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018890 2024-04-25T08:55:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2024-04-25T08:55:07.243 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018890 2024-04-25T08:55:07.243 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:55:07.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018890 -lt 128849018890 2024-04-25T08:55:07.676 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:55:07.676 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:07.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:08.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:55:08.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:55:08.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:55:08.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:55:08.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:55:08.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:55:08.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:55:08.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:55:08.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:55:08.615 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:55:08.615 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:08.615 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:09.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:55:09.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:55:09.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:55:09.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: seq 1 200 2024-04-25T08:55:09.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:09.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj1 /dev/null 2024-04-25T08:55:09.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:09.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj2 /dev/null 2024-04-25T08:55:09.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:09.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj3 /dev/null 2024-04-25T08:55:09.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:09.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj4 /dev/null 2024-04-25T08:55:09.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:09.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj5 /dev/null 2024-04-25T08:55:09.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:09.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj6 /dev/null 2024-04-25T08:55:10.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:10.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj7 /dev/null 2024-04-25T08:55:10.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:10.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj8 /dev/null 2024-04-25T08:55:10.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:10.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj9 /dev/null 2024-04-25T08:55:10.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:10.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj10 /dev/null 2024-04-25T08:55:10.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:10.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj11 /dev/null 2024-04-25T08:55:10.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:10.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj12 /dev/null 2024-04-25T08:55:10.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:10.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj13 /dev/null 2024-04-25T08:55:11.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:11.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj14 /dev/null 2024-04-25T08:55:11.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:11.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj15 /dev/null 2024-04-25T08:55:11.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:11.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj16 /dev/null 2024-04-25T08:55:11.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:11.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj17 /dev/null 2024-04-25T08:55:11.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:11.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj18 /dev/null 2024-04-25T08:55:11.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:11.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj19 /dev/null 2024-04-25T08:55:11.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:11.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj20 /dev/null 2024-04-25T08:55:12.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:12.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj21 /dev/null 2024-04-25T08:55:12.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:12.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj22 /dev/null 2024-04-25T08:55:12.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:12.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj23 /dev/null 2024-04-25T08:55:12.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:12.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj24 /dev/null 2024-04-25T08:55:12.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:12.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj25 /dev/null 2024-04-25T08:55:12.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:12.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj26 /dev/null 2024-04-25T08:55:12.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:12.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj27 /dev/null 2024-04-25T08:55:13.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:13.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj28 /dev/null 2024-04-25T08:55:13.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:13.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj29 /dev/null 2024-04-25T08:55:13.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:13.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj30 /dev/null 2024-04-25T08:55:13.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:13.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj31 /dev/null 2024-04-25T08:55:13.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:13.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj32 /dev/null 2024-04-25T08:55:13.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:13.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj33 /dev/null 2024-04-25T08:55:13.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:13.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj34 /dev/null 2024-04-25T08:55:14.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:14.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj35 /dev/null 2024-04-25T08:55:14.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:14.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj36 /dev/null 2024-04-25T08:55:14.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:14.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj37 /dev/null 2024-04-25T08:55:14.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:14.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj38 /dev/null 2024-04-25T08:55:14.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:14.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj39 /dev/null 2024-04-25T08:55:14.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:14.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj40 /dev/null 2024-04-25T08:55:14.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:14.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj41 /dev/null 2024-04-25T08:55:15.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:15.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj42 /dev/null 2024-04-25T08:55:15.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:15.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj43 /dev/null 2024-04-25T08:55:15.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:15.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj44 /dev/null 2024-04-25T08:55:15.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:15.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj45 /dev/null 2024-04-25T08:55:15.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:15.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj46 /dev/null 2024-04-25T08:55:15.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:15.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj47 /dev/null 2024-04-25T08:55:15.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:15.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj48 /dev/null 2024-04-25T08:55:16.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:16.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj49 /dev/null 2024-04-25T08:55:16.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:16.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj50 /dev/null 2024-04-25T08:55:16.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:16.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj51 /dev/null 2024-04-25T08:55:16.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:16.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj52 /dev/null 2024-04-25T08:55:16.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:16.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj53 /dev/null 2024-04-25T08:55:16.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:16.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj54 /dev/null 2024-04-25T08:55:16.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:16.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj55 /dev/null 2024-04-25T08:55:16.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:16.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj56 /dev/null 2024-04-25T08:55:17.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:17.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj57 /dev/null 2024-04-25T08:55:17.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:17.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj58 /dev/null 2024-04-25T08:55:17.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:17.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj59 /dev/null 2024-04-25T08:55:17.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:17.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj60 /dev/null 2024-04-25T08:55:17.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:17.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj61 /dev/null 2024-04-25T08:55:17.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:17.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj62 /dev/null 2024-04-25T08:55:17.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:17.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj63 /dev/null 2024-04-25T08:55:18.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:18.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj64 /dev/null 2024-04-25T08:55:18.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:18.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj65 /dev/null 2024-04-25T08:55:18.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:18.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj66 /dev/null 2024-04-25T08:55:18.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:18.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj67 /dev/null 2024-04-25T08:55:18.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:18.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj68 /dev/null 2024-04-25T08:55:18.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:18.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj69 /dev/null 2024-04-25T08:55:18.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:18.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj70 /dev/null 2024-04-25T08:55:18.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:18.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj71 /dev/null 2024-04-25T08:55:19.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:19.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj72 /dev/null 2024-04-25T08:55:19.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:19.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj73 /dev/null 2024-04-25T08:55:19.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:19.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj74 /dev/null 2024-04-25T08:55:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj75 /dev/null 2024-04-25T08:55:19.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:19.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj76 /dev/null 2024-04-25T08:55:19.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:19.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj77 /dev/null 2024-04-25T08:55:19.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:19.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj78 /dev/null 2024-04-25T08:55:20.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:20.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj79 /dev/null 2024-04-25T08:55:20.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:20.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj80 /dev/null 2024-04-25T08:55:20.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:20.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj81 /dev/null 2024-04-25T08:55:20.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:20.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj82 /dev/null 2024-04-25T08:55:20.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:20.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj83 /dev/null 2024-04-25T08:55:20.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:20.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj84 /dev/null 2024-04-25T08:55:20.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:20.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj85 /dev/null 2024-04-25T08:55:21.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:21.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj86 /dev/null 2024-04-25T08:55:21.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:21.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj87 /dev/null 2024-04-25T08:55:21.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:21.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj88 /dev/null 2024-04-25T08:55:21.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:21.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj89 /dev/null 2024-04-25T08:55:21.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:21.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj90 /dev/null 2024-04-25T08:55:21.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:21.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj91 /dev/null 2024-04-25T08:55:21.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:21.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj92 /dev/null 2024-04-25T08:55:22.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:22.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj93 /dev/null 2024-04-25T08:55:22.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:22.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj94 /dev/null 2024-04-25T08:55:22.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:22.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj95 /dev/null 2024-04-25T08:55:22.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:22.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj96 /dev/null 2024-04-25T08:55:22.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:22.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj97 /dev/null 2024-04-25T08:55:22.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:22.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj98 /dev/null 2024-04-25T08:55:22.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:22.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj99 /dev/null 2024-04-25T08:55:22.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:22.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj100 /dev/null 2024-04-25T08:55:23.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:23.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj101 /dev/null 2024-04-25T08:55:23.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:23.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj102 /dev/null 2024-04-25T08:55:23.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:23.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj103 /dev/null 2024-04-25T08:55:23.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:23.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj104 /dev/null 2024-04-25T08:55:23.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:23.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj105 /dev/null 2024-04-25T08:55:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj106 /dev/null 2024-04-25T08:55:23.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:23.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj107 /dev/null 2024-04-25T08:55:24.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:24.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj108 /dev/null 2024-04-25T08:55:24.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:24.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj109 /dev/null 2024-04-25T08:55:24.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:24.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj110 /dev/null 2024-04-25T08:55:24.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:24.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj111 /dev/null 2024-04-25T08:55:24.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:24.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj112 /dev/null 2024-04-25T08:55:24.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:24.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj113 /dev/null 2024-04-25T08:55:24.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:24.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj114 /dev/null 2024-04-25T08:55:25.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:25.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj115 /dev/null 2024-04-25T08:55:25.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:25.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj116 /dev/null 2024-04-25T08:55:25.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:25.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj117 /dev/null 2024-04-25T08:55:25.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:25.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj118 /dev/null 2024-04-25T08:55:25.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:25.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj119 /dev/null 2024-04-25T08:55:25.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:25.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj120 /dev/null 2024-04-25T08:55:25.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:25.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj121 /dev/null 2024-04-25T08:55:26.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:26.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj122 /dev/null 2024-04-25T08:55:26.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:26.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj123 /dev/null 2024-04-25T08:55:26.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:26.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj124 /dev/null 2024-04-25T08:55:26.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:26.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj125 /dev/null 2024-04-25T08:55:26.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:26.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj126 /dev/null 2024-04-25T08:55:26.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:26.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj127 /dev/null 2024-04-25T08:55:26.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:26.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj128 /dev/null 2024-04-25T08:55:27.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:27.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj129 /dev/null 2024-04-25T08:55:27.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:27.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj130 /dev/null 2024-04-25T08:55:27.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:27.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj131 /dev/null 2024-04-25T08:55:27.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:27.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj132 /dev/null 2024-04-25T08:55:27.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:27.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj133 /dev/null 2024-04-25T08:55:27.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:27.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj134 /dev/null 2024-04-25T08:55:27.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:27.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj135 /dev/null 2024-04-25T08:55:28.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:28.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj136 /dev/null 2024-04-25T08:55:28.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:28.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj137 /dev/null 2024-04-25T08:55:28.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:28.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj138 /dev/null 2024-04-25T08:55:28.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:28.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj139 /dev/null 2024-04-25T08:55:28.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:28.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj140 /dev/null 2024-04-25T08:55:28.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:28.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj141 /dev/null 2024-04-25T08:55:28.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:28.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj142 /dev/null 2024-04-25T08:55:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj143 /dev/null 2024-04-25T08:55:29.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:29.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj144 /dev/null 2024-04-25T08:55:29.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:29.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj145 /dev/null 2024-04-25T08:55:29.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:29.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj146 /dev/null 2024-04-25T08:55:29.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:29.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj147 /dev/null 2024-04-25T08:55:29.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:29.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj148 /dev/null 2024-04-25T08:55:29.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:29.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj149 /dev/null 2024-04-25T08:55:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj150 /dev/null 2024-04-25T08:55:30.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:30.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj151 /dev/null 2024-04-25T08:55:30.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:30.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj152 /dev/null 2024-04-25T08:55:30.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:30.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj153 /dev/null 2024-04-25T08:55:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj154 /dev/null 2024-04-25T08:55:30.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:30.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj155 /dev/null 2024-04-25T08:55:30.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:30.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj156 /dev/null 2024-04-25T08:55:30.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:30.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj157 /dev/null 2024-04-25T08:55:30.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:30.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj158 /dev/null 2024-04-25T08:55:31.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:31.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj159 /dev/null 2024-04-25T08:55:31.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:31.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj160 /dev/null 2024-04-25T08:55:31.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:31.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj161 /dev/null 2024-04-25T08:55:31.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:31.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj162 /dev/null 2024-04-25T08:55:31.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:31.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj163 /dev/null 2024-04-25T08:55:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:31.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj164 /dev/null 2024-04-25T08:55:31.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:31.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj165 /dev/null 2024-04-25T08:55:32.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:32.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj166 /dev/null 2024-04-25T08:55:32.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:32.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj167 /dev/null 2024-04-25T08:55:32.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:32.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj168 /dev/null 2024-04-25T08:55:32.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:32.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj169 /dev/null 2024-04-25T08:55:32.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:32.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj170 /dev/null 2024-04-25T08:55:32.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:32.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj171 /dev/null 2024-04-25T08:55:32.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:32.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj172 /dev/null 2024-04-25T08:55:32.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:32.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj173 /dev/null 2024-04-25T08:55:33.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:33.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj174 /dev/null 2024-04-25T08:55:33.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:33.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj175 /dev/null 2024-04-25T08:55:33.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:33.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj176 /dev/null 2024-04-25T08:55:33.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:33.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj177 /dev/null 2024-04-25T08:55:33.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:33.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj178 /dev/null 2024-04-25T08:55:33.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:33.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj179 /dev/null 2024-04-25T08:55:34.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:34.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj180 /dev/null 2024-04-25T08:55:34.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:34.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj181 /dev/null 2024-04-25T08:55:34.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:34.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj182 /dev/null 2024-04-25T08:55:34.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:34.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj183 /dev/null 2024-04-25T08:55:34.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:34.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj184 /dev/null 2024-04-25T08:55:34.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:34.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj185 /dev/null 2024-04-25T08:55:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:34.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj186 /dev/null 2024-04-25T08:55:34.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:34.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj187 /dev/null 2024-04-25T08:55:35.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:35.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj188 /dev/null 2024-04-25T08:55:35.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:35.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj189 /dev/null 2024-04-25T08:55:35.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:35.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj190 /dev/null 2024-04-25T08:55:35.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:35.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj191 /dev/null 2024-04-25T08:55:35.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:35.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj192 /dev/null 2024-04-25T08:55:35.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:35.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj193 /dev/null 2024-04-25T08:55:35.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:35.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj194 /dev/null 2024-04-25T08:55:36.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:36.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj195 /dev/null 2024-04-25T08:55:36.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:36.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj196 /dev/null 2024-04-25T08:55:36.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:36.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj197 /dev/null 2024-04-25T08:55:36.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:36.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj198 /dev/null 2024-04-25T08:55:36.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:36.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj199 /dev/null 2024-04-25T08:55:36.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:140: do_recovery_out1: for i in $(seq 1 $objects) 2024-04-25T08:55:36.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:142: do_recovery_out1: rados -p test put obj200 /dev/null 2024-04-25T08:55:36.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: get_primary test obj1 2024-04-25T08:55:36.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:55:36.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:55:36.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:55:36.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:55:37.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:145: do_recovery_out1: local primary=1 2024-04-25T08:55:37.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: get_pg test obj1 2024-04-25T08:55:37.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T08:55:37.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T08:55:37.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T08:55:37.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T08:55:37.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:146: do_recovery_out1: local PG=1.0 2024-04-25T08:55:37.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: get_not_primary test obj1 2024-04-25T08:55:37.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T08:55:37.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T08:55:37.618 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T08:55:37.618 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:55:37.618 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:55:37.618 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:55:37.619 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:55:38.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T08:55:38.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T08:55:38.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T08:55:38.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:148: do_recovery_out1: local otherosd=0 2024-04-25T08:55:38.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:150: do_recovery_out1: ceph osd set norecover 2024-04-25T08:55:39.386 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is set 2024-04-25T08:55:39.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: cat td/osd-recovery-stats/osd.0.pid 2024-04-25T08:55:39.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:151: do_recovery_out1: kill 329240 2024-04-25T08:55:39.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:152: do_recovery_out1: ceph osd down osd.0 2024-04-25T08:55:40.005 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T08:55:40.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:153: do_recovery_out1: ceph osd out osd.0 2024-04-25T08:55:40.585 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T08:55:40.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:154: do_recovery_out1: ceph osd unset norecover 2024-04-25T08:55:41.698 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is unset 2024-04-25T08:55:41.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: get_primary test obj1 2024-04-25T08:55:41.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T08:55:41.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T08:55:41.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T08:55:41.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T08:55:42.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:155: do_recovery_out1: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T08:55:42.346 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T08:55:42.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:156: do_recovery_out1: sleep 2 2024-04-25T08:55:44.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:158: do_recovery_out1: wait_for_clean 2024-04-25T08:55:44.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:55:44.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:55:44.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:55:44.362 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:55:44.363 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:55:44.363 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:55:44.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:55:44.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:55:44.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:55:44.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:55:44.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:55:44.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:55:44.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:55:44.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:55:44.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:55:44.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:55:44.918 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:55:44.919 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:55:44.919 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:55:44.919 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T08:55:44.919 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T08:55:44.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:55:44.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:44.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:55:45.102 INFO:tasks.workunit.client.0.smithi134.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-04-25T08:55:45.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq= 2024-04-25T08:55:45.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z '' 2024-04-25T08:55:45.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2221: flush_pg_stats: continue 2024-04-25T08:55:45.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:45.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:55:45.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673009 2024-04-25T08:55:45.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673009 2024-04-25T08:55:45.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673009' 2024-04-25T08:55:45.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:45.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:55:45.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509479 2024-04-25T08:55:45.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509479 2024-04-25T08:55:45.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673009 2-64424509479' 2024-04-25T08:55:45.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:45.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:55:45.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345952 2024-04-25T08:55:45.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345952 2024-04-25T08:55:45.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673009 2-64424509479 3-85899345952' 2024-04-25T08:55:45.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:45.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:55:45.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182425 2024-04-25T08:55:45.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182425 2024-04-25T08:55:45.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673009 2-64424509479 3-85899345952 4-107374182425' 2024-04-25T08:55:45.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:55:45.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:55:45.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018900 2024-04-25T08:55:45.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018900 2024-04-25T08:55:45.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 1-42949673009 2-64424509479 3-85899345952 4-107374182425 5-128849018900' 2024-04-25T08:55:45.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:45.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673009 2024-04-25T08:55:45.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:45.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:55:45.893 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673009 2024-04-25T08:55:45.893 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:45.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673009 2024-04-25T08:55:45.894 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673009 2024-04-25T08:55:45.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673009' 2024-04-25T08:55:45.894 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:55:46.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673007 -lt 42949673009 2024-04-25T08:55:46.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T08:55:47.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T08:55:47.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:55:47.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673009 -lt 42949673009 2024-04-25T08:55:47.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:47.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:47.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509479 2024-04-25T08:55:47.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:55:47.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509479 2024-04-25T08:55:47.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:47.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509479 2024-04-25T08:55:47.791 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509479 2024-04-25T08:55:47.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509479' 2024-04-25T08:55:47.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:55:48.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509480 -lt 64424509479 2024-04-25T08:55:48.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:48.271 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345952 2024-04-25T08:55:48.271 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:48.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:55:48.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345952 2024-04-25T08:55:48.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:48.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345952 2024-04-25T08:55:48.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345952' 2024-04-25T08:55:48.275 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345952 2024-04-25T08:55:48.275 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:55:48.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345952 -lt 85899345952 2024-04-25T08:55:48.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:48.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:48.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182425 2024-04-25T08:55:48.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:55:48.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:48.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182425 2024-04-25T08:55:48.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182425 2024-04-25T08:55:48.753 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182425 2024-04-25T08:55:48.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182425' 2024-04-25T08:55:48.754 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:55:49.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182426 -lt 107374182425 2024-04-25T08:55:49.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:55:49.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018900 2024-04-25T08:55:49.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:55:49.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:55:49.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018900 2024-04-25T08:55:49.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:55:49.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018900 2024-04-25T08:55:49.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018900' 2024-04-25T08:55:49.265 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018900 2024-04-25T08:55:49.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T08:55:49.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018901 -lt 128849018900 2024-04-25T08:55:49.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T08:55:49.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:49.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:50.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T08:55:50.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:55:50.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:55:50.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:55:50.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:55:50.381 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:55:50.381 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:55:50.381 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:55:50.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:55:50.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:55:50.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:50.873 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:51.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:55:51.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T08:55:51.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T08:55:51.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T08:55:51.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:55:51.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:55:51.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:55:51.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:55:51.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:55:51.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:55:51.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:55:51.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:55:51.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:55:51.650 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:55:52.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:55:52.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:55:52.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:52.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:52.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:55:52.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:55:52.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:55:52.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:55:52.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:55:52.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:55:52.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:55:52.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:55:52.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:55:53.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-04-25T08:55:53.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-04-25T08:55:53.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:55:53.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:55:53.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:55:53.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:55:53.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:55:53.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:55:53.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:55:53.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:55:53.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:55:53.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:55:53.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:55:53.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:55:53.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:55:53.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:53.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:54.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:55:54.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:55:54.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:55:54.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:55:54.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:55:54.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:55:54.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:55:54.459 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:55:54.460 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:55:55.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=13 2024-04-25T08:55:55.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 13 '!=' null 2024-04-25T08:55:55.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:55:55.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:55:55.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:55:55.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:55:55.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:55:55.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:55:55.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:55:55.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:55:55.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:55:55.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:55:55.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:55:55.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:55:55.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:55:55.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:55.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:56.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:55:56.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:55:56.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:55:56.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:55:56.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:55:56.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:55:56.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:55:56.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:55:56.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:55:56.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:55:56.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:55:56.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:55:56.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:55:56.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:55:56.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:55:56.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:55:56.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:55:56.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:55:56.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:55:56.902 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:55:56.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:55:56.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:55:57.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T08:55:57.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:55:57.336 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:57.336 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:57.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T08:55:57.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T08:55:57.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T08:55:57.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T08:55:57.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T08:55:57.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T08:55:57.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T08:55:57.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T08:55:57.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T08:55:58.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=11 2024-04-25T08:55:58.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 11 '!=' null 2024-04-25T08:55:58.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T08:55:58.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T08:55:58.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T08:55:58.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T08:55:58.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T08:55:58.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T08:55:58.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T08:55:58.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T08:55:58.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T08:55:58.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T08:55:58.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T08:55:58.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T08:55:58.994 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T08:55:58.994 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T08:55:58.995 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T08:55:59.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:160: do_recovery_out1: check td/osd-recovery-stats 1.0 1 replicated 200 0 0 0 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=200 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T08:55:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T08:55:59.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T08:55:59.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:59.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T08:55:59.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T08:55:59.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T08:55:59.516 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:59.516 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T08:55:59.516 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T08:55:59.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2,4]/[1,2]' 2024-04-25T08:55:59.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T08:55:59.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:55:59.527 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:59.527 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2,4]/[1,2] ' 2024-04-25T08:55:59.527 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T08:55:59.527 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T08:55:59.528 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:55:59.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=200 2024-04-25T08:55:59.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 200 200 2024-04-25T08:55:59.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T08:55:59.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:55:59.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T08:55:59.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:55:59.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:55:59.535 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:59.535 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T08:55:59.535 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:55:59.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T08:55:59.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T08:55:59.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:55:59.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:55:59.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:55:59.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:55:59.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T08:55:59.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:55:59.546 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:59.546 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2,4]/[1,2] ' 2024-04-25T08:55:59.546 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T08:55:59.546 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T08:55:59.546 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:55:59.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T08:55:59.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T08:55:59.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T08:55:59.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T08:55:59.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T08:55:59.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T08:55:59.552 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T08:55:59.553 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T08:55:59.553 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T08:55:59.553 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T08:55:59.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T08:55:59.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T08:55:59.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T08:55:59.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T08:55:59.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T08:55:59.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T08:55:59.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T08:55:59.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:162: do_recovery_out1: delete_pool test 2024-04-25T08:55:59.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T08:55:59.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T08:56:00.050 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T08:56:00.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:163: do_recovery_out1: kill_daemons td/osd-recovery-stats 2024-04-25T08:56:00.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:56:00.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:56:00.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:56:00.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:56:00.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:56:05.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:56:05.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T08:56:05.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:56:05.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:56:05.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:56:05.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:56:05.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:56:05.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:56:05.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:56:05.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:56:05.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:56:05.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:56:05.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:56:05.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:56:05.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:56:05.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:56:05.400 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:56:05.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:56:05.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:56:05.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:56:05.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:56:05.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:56:05.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:56:05.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:56:05.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:56:05.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:05.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:56:05.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T08:56:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:56:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:56:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T08:56:05.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T08:56:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T08:56:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T08:56:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T08:56:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T08:56:05.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T08:56:05.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T08:56:05.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T08:56:05.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T08:56:05.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T08:56:05.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T08:56:05.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T08:56:05.451 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T08:56:05.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T08:56:05.451 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T08:56:05.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T08:56:05.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T08:56:05.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T08:56:05.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:56:05.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T08:56:05.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T08:56:05.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T08:56:05.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T08:56:05.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T08:56:05.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T08:56:05.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T08:56:05.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:05.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:56:05.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T08:56:05.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T08:56:05.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T08:56:05.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T08:56:05.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T08:56:05.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:05.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:56:05.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.280166 2024-04-25T08:56:05.460 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T08:56:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T08:56:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T08:56:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T08:56:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizedown td/osd-recovery-stats 2024-04-25T08:56:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:241: TEST_recovery_sizedown: local dir=td/osd-recovery-stats 2024-04-25T08:56:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:243: TEST_recovery_sizedown: run_mon td/osd-recovery-stats a 2024-04-25T08:56:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T08:56:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T08:56:05.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T08:56:05.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T08:56:05.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T08:56:05.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T08:56:05.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T08:56:05.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:56:05.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:56:05.951 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:56:05.951 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:05.951 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:56:05.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:56:05.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T08:56:06.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T08:56:06.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T08:56:06.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:56:06.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:56:06.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T08:56:06.165 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T08:56:06.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:56:06.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:56:06.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:56:06.165 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:56:06.166 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:06.166 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:56:06.166 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:56:06.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:56:06.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get fsid 2024-04-25T08:56:06.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T08:56:06.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T08:56:06.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T08:56:06.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T08:56:06.278 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T08:56:06.278 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T08:56:06.278 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T08:56:06.278 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T08:56:06.279 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T08:56:06.279 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:06.279 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:56:06.279 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T08:56:06.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T08:56:06.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get mon_host 2024-04-25T08:56:06.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:244: TEST_recovery_sizedown: run_mgr td/osd-recovery-stats x 2024-04-25T08:56:06.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T08:56:06.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T08:56:06.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T08:56:06.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T08:56:06.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T08:56:06.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:56:07.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:56:07.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T08:56:07.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:245: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 0 2024-04-25T08:56:07.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:56:07.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:56:07.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:56:07.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:56:07.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:56:07.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:56:07.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T08:56:07.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:56:07.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=72372d9a-3814-485f-8a00-9d2849783725 2024-04-25T08:56:07.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 72372d9a-3814-485f-8a00-9d2849783725' 2024-04-25T08:56:07.342 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 72372d9a-3814-485f-8a00-9d2849783725 2024-04-25T08:56:07.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:56:07.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCnGipmr1D+FRAAUIyUTaPnBZ28F7dN+O0EOg== 2024-04-25T08:56:07.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCnGipmr1D+FRAAUIyUTaPnBZ28F7dN+O0EOg=="}' 2024-04-25T08:56:07.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 72372d9a-3814-485f-8a00-9d2849783725 -i td/osd-recovery-stats/0/new.json 2024-04-25T08:56:07.707 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:56:07.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T08:56:07.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCnGipmr1D+FRAAUIyUTaPnBZ28F7dN+O0EOg== --osd-uuid 72372d9a-3814-485f-8a00-9d2849783725 2024-04-25T08:56:07.753 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:07.760+0000 7fa0947617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:07.783 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:07.792+0000 7fa0947617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:07.785 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:07.792+0000 7fa0947617c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:07.786 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:07.796+0000 7fa0947617c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:56:07.786 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:07.796+0000 7fa0947617c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:56:07.786 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:07.796+0000 7fa0947617c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T08:56:07.824 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:07.832+0000 7fa0947617c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T08:56:14.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T08:56:14.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:56:14.228 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T08:56:14.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T08:56:14.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:56:14.792 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T08:56:14.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T08:56:14.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:56:14.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:56:14.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:56:14.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:56:14.824 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:14.832+0000 7f86b1b257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:14.826 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:14.836+0000 7f86b1b257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:14.827 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:14.836+0000 7f86b1b257c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:15.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T08:56:15.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:56:15.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T08:56:15.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:56:15.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:56:15.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:15.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:56:15.208 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:56:15.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:15.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:15.371 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:15.380+0000 7f86b1b257c0 -1 Falling back to public interface 2024-04-25T08:56:15.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:16.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:16.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:16.666 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:56:16.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:56:16.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:16.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:17.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:18.007 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:18.016+0000 7f86b1b257c0 -1 osd.0 0 log_to_monitors true 2024-04-25T08:56:18.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:18.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:18.074 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:56:18.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:56:18.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:18.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:18.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:19.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:19.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:19.741 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:56:19.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:56:19.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:19.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:20.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:21.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:21.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:21.182 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:56:21.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:56:21.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:21.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:21.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:22.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:22.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:22.629 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:56:22.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:56:22.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:22.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:23.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:24.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:24.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:24.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:56:24.147 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:56:24.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:24.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:24.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:25.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:25.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:25.600 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:56:25.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:56:25.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:25.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:26.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:27.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:27.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:27.114 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:56:27.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:56:27.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:27.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:27.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:28.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:28.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:28.569 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:56:28.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:56:28.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:28.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:29.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:30.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:30.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:30.018 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:56:30.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:56:30.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:30.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:30.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:31.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:31.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:31.455 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:56:31.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:56:31.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:31.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:32.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:33.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:33.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:33.666 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:56:33.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:56:33.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:33.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:34.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:35.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:35.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:35.435 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:56:35.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:56:35.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:35.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:36.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:37.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:37.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:37.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:56:37.319 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:56:37.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:37.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:38.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:39.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:39.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:39.161 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:56:39.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:56:39.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:39.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:40.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:41.004 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:56:41.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:56:41.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:41.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:41.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:42.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:42.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:42.863 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:56:42.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:56:42.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:42.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T08:56:43.309 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3267229571,v1:127.0.0.1:6803/3267229571] [v2:127.0.0.1:6804/3267229571,v1:127.0.0.1:6805/3267229571] exists,up 72372d9a-3814-485f-8a00-9d2849783725 2024-04-25T08:56:43.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:56:43.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:56:43.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:56:43.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:246: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 1 2024-04-25T08:56:43.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:56:43.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:56:43.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:56:43.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:56:43.314 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:56:43.314 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:56:43.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:56:43.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T08:56:43.316 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:56:43.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5d9d2ac0-3f25-4f51-9948-64495cf37da1 2024-04-25T08:56:43.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5d9d2ac0-3f25-4f51-9948-64495cf37da1' 2024-04-25T08:56:43.316 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 5d9d2ac0-3f25-4f51-9948-64495cf37da1 2024-04-25T08:56:43.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:56:43.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDLGipmoTu8FBAA6Lpq853uhdFGfwLWEwZbsw== 2024-04-25T08:56:43.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDLGipmoTu8FBAA6Lpq853uhdFGfwLWEwZbsw=="}' 2024-04-25T08:56:43.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5d9d2ac0-3f25-4f51-9948-64495cf37da1 -i td/osd-recovery-stats/1/new.json 2024-04-25T08:56:43.844 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:56:43.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T08:56:43.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDLGipmoTu8FBAA6Lpq853uhdFGfwLWEwZbsw== --osd-uuid 5d9d2ac0-3f25-4f51-9948-64495cf37da1 2024-04-25T08:56:43.895 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:43.903+0000 7f66aed817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:43.955 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:43.963+0000 7f66aed817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:43.957 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:43.963+0000 7f66aed817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:43.958 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:43.967+0000 7f66aed817c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:56:43.958 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:43.967+0000 7f66aed817c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:56:43.958 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:43.967+0000 7f66aed817c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T08:56:44.080 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:44.087+0000 7f66aed817c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T08:56:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T08:56:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:56:50.582 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T08:56:50.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T08:56:50.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:56:51.155 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T08:56:51.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T08:56:51.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:56:51.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:56:51.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:56:51.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:56:51.186 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:51.195+0000 7f3cdf8377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:51.189 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:51.195+0000 7f3cdf8377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:51.190 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:51.199+0000 7f3cdf8377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:56:51.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T08:56:51.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:56:51.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T08:56:51.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:56:51.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:56:51.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:51.555 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:56:51.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:56:51.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:51.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:51.736 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:51.743+0000 7f3cdf8377c0 -1 Falling back to public interface 2024-04-25T08:56:51.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:52.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:52.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:52.972 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:56:52.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:56:52.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:52.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:53.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:54.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:54.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:54.377 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:56:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:56:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:54.399 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:56:54.407+0000 7f3cdf8377c0 -1 osd.1 0 log_to_monitors true 2024-04-25T08:56:54.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:55.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:55.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:55.953 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:56:55.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:56:55.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:55.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:56.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:57.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:57.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:57.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:56:57.543 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:56:57.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:57.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:58.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:56:59.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:56:59.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:56:59.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:56:59.012 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:56:59.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:56:59.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:56:59.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:00.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:00.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:00.488 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:57:00.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:57:00.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:00.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:00.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:01.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:01.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:01.990 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:57:01.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:57:01.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:01.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:02.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:03.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:03.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:03.547 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:57:03.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:57:03.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:03.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:03.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:04.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:04.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:04.976 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:57:04.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:57:04.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:04.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:05.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:06.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:06.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:06.465 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:57:06.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:57:06.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:06.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:07.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:08.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:08.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:08.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:57:08.059 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:57:08.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:08.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:08.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:09.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:09.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:09.514 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:57:09.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:57:09.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:09.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:10.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:11.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:11.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:11.041 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:57:11.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:57:11.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:11.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:11.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:12.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:12.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:12.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:57:12.689 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:57:12.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:12.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:13.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:14.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:14.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:14.184 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:57:14.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:57:14.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:14.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:15.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:16.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:16.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:16.390 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:57:16.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:57:16.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:16.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:17.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:18.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:18.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:18.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:57:18.529 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:57:18.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:18.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:19.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:20.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:20.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:20.268 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:57:20.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:57:20.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:20.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T08:57:20.815 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1328651299,v1:127.0.0.1:6811/1328651299] [v2:127.0.0.1:6812/1328651299,v1:127.0.0.1:6813/1328651299] exists,up 5d9d2ac0-3f25-4f51-9948-64495cf37da1 2024-04-25T08:57:20.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:57:20.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:57:20.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:57:20.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:247: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 2 2024-04-25T08:57:20.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:57:20.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:57:20.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:57:20.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:57:20.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:57:20.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:57:20.819 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:57:20.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T08:57:20.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:57:20.821 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 89450c2e-7fab-4ec2-b575-0bd8955f2396 2024-04-25T08:57:20.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89450c2e-7fab-4ec2-b575-0bd8955f2396 2024-04-25T08:57:20.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 89450c2e-7fab-4ec2-b575-0bd8955f2396' 2024-04-25T08:57:20.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:57:20.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDwGipmWMznMhAA8zkHVVnX2vFfAwnU7G33oA== 2024-04-25T08:57:20.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDwGipmWMznMhAA8zkHVVnX2vFfAwnU7G33oA=="}' 2024-04-25T08:57:20.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89450c2e-7fab-4ec2-b575-0bd8955f2396 -i td/osd-recovery-stats/2/new.json 2024-04-25T08:57:21.358 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:57:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T08:57:21.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDwGipmWMznMhAA8zkHVVnX2vFfAwnU7G33oA== --osd-uuid 89450c2e-7fab-4ec2-b575-0bd8955f2396 2024-04-25T08:57:21.409 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:21.415+0000 7f0ed2b407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:21.444 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:21.451+0000 7f0ed2b407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:21.445 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:21.451+0000 7f0ed2b407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:21.445 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:21.451+0000 7f0ed2b407c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:57:21.445 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:21.451+0000 7f0ed2b407c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:57:21.445 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:21.451+0000 7f0ed2b407c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T08:57:21.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:21.491+0000 7f0ed2b407c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T08:57:28.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T08:57:28.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:57:28.017 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T08:57:28.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T08:57:28.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:57:28.565 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T08:57:28.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T08:57:28.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:57:28.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:57:28.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:57:28.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:57:28.597 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:28.602+0000 7fe555f5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:28.625 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:28.630+0000 7fe555f5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:28.626 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:28.634+0000 7fe555f5d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:29.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T08:57:29.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:57:29.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T08:57:29.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:57:29.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:57:29.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:29.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:57:29.014 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:57:29.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:29.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:29.175 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:29.182+0000 7fe555f5d7c0 -1 Falling back to public interface 2024-04-25T08:57:29.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:30.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:30.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:30.450 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:57:30.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:57:30.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:30.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:30.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:31.849 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:31.854+0000 7fe555f5d7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T08:57:31.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:31.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:31.864 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:57:31.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:57:31.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:31.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:32.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:33.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:33.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:33.618 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:57:33.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:57:33.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:33.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:34.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:35.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:35.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:35.044 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:57:35.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:57:35.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:35.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:35.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:36.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:36.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:36.482 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:57:36.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:57:36.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:36.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:36.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:38.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:38.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:38.001 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:57:38.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:57:38.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:38.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:38.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:39.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:39.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:39.461 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:57:39.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:57:39.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:39.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:39.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:40.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:40.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:40.985 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:57:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:57:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:41.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:42.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:42.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:42.463 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:57:42.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:57:42.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:42.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:43.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:44.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:44.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:44.012 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:57:44.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:57:44.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:44.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:44.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:45.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:45.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:45.486 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:57:45.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:57:45.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:45.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:46.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:47.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:47.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:47.137 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:57:47.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:57:47.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:47.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:48.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:49.421 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:57:49.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:57:49.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:49.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:50.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:51.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:51.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:51.259 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:57:51.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:57:51.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:51.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:52.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:53.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:53.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:53.162 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:57:53.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:57:53.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:53.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:53.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:54.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:54.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:54.970 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:57:54.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:57:54.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:54.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:56.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:57.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:57.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:57.055 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:57:57.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:57:57.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:57.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:57.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:57:58.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:57:58.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:57:58.702 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:57:58.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:57:58.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:57:58.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T08:57:59.143 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/158804402,v1:127.0.0.1:6819/158804402] [v2:127.0.0.1:6820/158804402,v1:127.0.0.1:6821/158804402] exists,up 89450c2e-7fab-4ec2-b575-0bd8955f2396 2024-04-25T08:57:59.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:57:59.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:57:59.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:57:59.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:248: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 3 2024-04-25T08:57:59.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:57:59.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:57:59.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:57:59.147 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:57:59.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:57:59.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:57:59.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T08:57:59.149 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:57:59.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=35459726-3f4e-4047-acb6-bbed57320551 2024-04-25T08:57:59.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 35459726-3f4e-4047-acb6-bbed57320551' 2024-04-25T08:57:59.150 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 35459726-3f4e-4047-acb6-bbed57320551 2024-04-25T08:57:59.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:57:59.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAXGypm0gOBChAAoC14xAYQT+wiRkG13ry74w== 2024-04-25T08:57:59.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAXGypm0gOBChAAoC14xAYQT+wiRkG13ry74w=="}' 2024-04-25T08:57:59.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 35459726-3f4e-4047-acb6-bbed57320551 -i td/osd-recovery-stats/3/new.json 2024-04-25T08:57:59.707 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:57:59.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T08:57:59.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXGypm0gOBChAAoC14xAYQT+wiRkG13ry74w== --osd-uuid 35459726-3f4e-4047-acb6-bbed57320551 2024-04-25T08:57:59.754 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:59.762+0000 7f3b7646f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:59.765 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:59.770+0000 7f3b7646f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:59.767 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:59.774+0000 7f3b7646f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:57:59.767 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:59.774+0000 7f3b7646f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:57:59.767 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:59.774+0000 7f3b7646f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:57:59.767 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:59.774+0000 7f3b7646f7c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T08:57:59.805 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:57:59.810+0000 7f3b7646f7c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T08:58:06.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T08:58:06.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:58:06.461 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T08:58:06.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T08:58:06.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:58:07.049 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T08:58:07.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T08:58:07.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:58:07.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:58:07.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:58:07.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:58:07.081 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:07.090+0000 7f90509297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:07.083 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:07.094+0000 7f90509297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:07.084 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:07.094+0000 7f90509297c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:07.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T08:58:07.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:58:07.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T08:58:07.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:58:07.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:58:07.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:07.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:58:07.501 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:58:07.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:07.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:07.635 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:07.646+0000 7f90509297c0 -1 Falling back to public interface 2024-04-25T08:58:07.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:08.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:08.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:08.943 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:58:08.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:58:08.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:08.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:09.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:10.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:10.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:10.348 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:58:10.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:58:10.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:10.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:10.370 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:10.381+0000 7f90509297c0 -1 osd.3 0 log_to_monitors true 2024-04-25T08:58:10.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:11.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:11.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:11.930 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:58:11.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:58:11.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:11.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:12.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:13.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:13.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:13.504 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:58:13.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:58:13.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:13.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:13.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:14.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:14.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:14.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:58:14.945 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:58:14.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:14.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:15.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:16.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:16.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:16.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:58:16.429 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:58:16.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:16.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:17.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:18.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:18.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:18.054 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:58:18.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:58:18.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:18.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:18.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:19.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:19.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:19.559 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:58:19.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:58:19.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:19.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:19.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:20.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:20.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:20.993 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:58:20.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:58:20.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:20.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:21.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:22.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:22.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:22.587 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:58:22.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:58:22.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:22.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:23.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:24.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:24.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:24.184 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:58:24.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:58:24.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:24.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:25.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:26.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:26.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:26.034 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:58:26.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:58:26.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:26.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:26.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:27.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:27.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:27.844 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:58:27.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:58:27.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:27.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:28.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:29.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:29.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:29.809 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:58:29.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:58:29.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:29.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:30.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:31.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:31.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:31.515 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:58:31.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:58:31.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:31.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:32.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:33.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:33.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:33.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:58:33.318 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:58:33.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:33.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:34.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:35.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:35.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:35.107 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:58:35.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:58:35.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:35.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:36.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:37.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:37.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:37.060 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:58:37.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:58:37.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:37.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T08:58:37.621 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4125111095,v1:127.0.0.1:6827/4125111095] [v2:127.0.0.1:6828/4125111095,v1:127.0.0.1:6829/4125111095] exists,up 35459726-3f4e-4047-acb6-bbed57320551 2024-04-25T08:58:37.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:58:37.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:58:37.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:58:37.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:249: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 4 2024-04-25T08:58:37.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:58:37.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:58:37.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T08:58:37.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:58:37.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:58:37.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:58:37.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:58:37.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:58:37.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:58:37.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T08:58:37.628 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:58:37.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d4929b5d-7695-4f23-bc2c-9ddd75a1a4f5 2024-04-25T08:58:37.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 d4929b5d-7695-4f23-bc2c-9ddd75a1a4f5' 2024-04-25T08:58:37.629 INFO:tasks.workunit.client.0.smithi134.stdout:add osd4 d4929b5d-7695-4f23-bc2c-9ddd75a1a4f5 2024-04-25T08:58:37.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:58:37.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9GypmZmZIJxAA8oJFeiwmSXy2EAnJRlrPww== 2024-04-25T08:58:37.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9GypmZmZIJxAA8oJFeiwmSXy2EAnJRlrPww=="}' 2024-04-25T08:58:37.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d4929b5d-7695-4f23-bc2c-9ddd75a1a4f5 -i td/osd-recovery-stats/4/new.json 2024-04-25T08:58:38.291 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:58:38.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T08:58:38.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9GypmZmZIJxAA8oJFeiwmSXy2EAnJRlrPww== --osd-uuid d4929b5d-7695-4f23-bc2c-9ddd75a1a4f5 2024-04-25T08:58:38.341 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:38.349+0000 7fc6229bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:38.343 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:38.353+0000 7fc6229bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:38.344 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:38.353+0000 7fc6229bc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:38.345 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:38.353+0000 7fc6229bc7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:58:38.345 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:38.353+0000 7fc6229bc7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:58:38.345 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:38.353+0000 7fc6229bc7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T08:58:38.422 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:38.433+0000 7fc6229bc7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T08:58:45.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T08:58:45.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:58:45.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T08:58:45.170 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd4 key to auth repository 2024-04-25T08:58:45.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:58:45.726 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.4 2024-04-25T08:58:45.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T08:58:45.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:58:45.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:58:45.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:58:45.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:58:45.758 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:45.769+0000 7f8c3daba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:45.761 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:45.769+0000 7f8c3daba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:45.762 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:45.773+0000 7f8c3daba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:58:46.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T08:58:46.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:58:46.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T08:58:46.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:58:46.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:58:46.125 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:58:46.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:46.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:58:46.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:46.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:58:46.319 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:46.329+0000 7f8c3daba7c0 -1 Falling back to public interface 2024-04-25T08:58:46.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:47.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:47.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:47.566 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:58:47.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:58:47.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:47.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:58:47.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:48.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:48.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:48.971 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:58:48.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:58:48.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:48.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:58:49.054 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:58:49.065+0000 7f8c3daba7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T08:58:49.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:50.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:50.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:50.475 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:58:50.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:58:50.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:50.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:58:51.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:52.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:52.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:52.171 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:58:52.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:58:52.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:52.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:58:52.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:53.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:53.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:53.688 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:58:53.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:58:53.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:53.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:58:54.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:55.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:55.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:55.301 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:58:55.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:58:55.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:55.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:58:55.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:56.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:56.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:56.740 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:58:56.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:58:56.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:56.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:58:57.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:58.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:58.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:58.253 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:58:58.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:58:58.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:58.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:58:58.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:58:59.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:58:59.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:58:59.748 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:58:59.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:58:59.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:58:59.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:00.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:01.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:01.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:01.354 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:59:01.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:59:01.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:01.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:01.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:02.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:02.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:02.783 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:59:02.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:59:02.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:02.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:03.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:04.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:04.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:04.225 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:59:04.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:59:04.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:04.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:05.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:06.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:06.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:06.172 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:59:06.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:59:06.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:06.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:08.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:08.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:08.242 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:59:08.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:59:08.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:08.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:08.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:09.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:09.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:09.892 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:59:09.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:59:09.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:09.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:11.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:12.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:12.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:12.008 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:59:12.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:59:12.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:12.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:13.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:14.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:14.319 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:59:14.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:59:14.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:14.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:14.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:15.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:15.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:15.929 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T08:59:15.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T08:59:15.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:15.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T08:59:16.359 INFO:tasks.workunit.client.0.smithi134.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/174007220,v1:127.0.0.1:6835/174007220] [v2:127.0.0.1:6836/174007220,v1:127.0.0.1:6837/174007220] exists,up d4929b5d-7695-4f23-bc2c-9ddd75a1a4f5 2024-04-25T08:59:16.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:59:16.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:250: TEST_recovery_sizedown: run_osd td/osd-recovery-stats 5 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T08:59:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T08:59:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fe71518e-7b80-494f-b71d-fc3c2f3dadd1 2024-04-25T08:59:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 fe71518e-7b80-494f-b71d-fc3c2f3dadd1' 2024-04-25T08:59:16.371 INFO:tasks.workunit.client.0.smithi134.stdout:add osd5 fe71518e-7b80-494f-b71d-fc3c2f3dadd1 2024-04-25T08:59:16.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T08:59:16.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBkGypmSJnTFxAAuQniML2LNLWPuUgeqEjMWQ== 2024-04-25T08:59:16.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBkGypmSJnTFxAAuQniML2LNLWPuUgeqEjMWQ=="}' 2024-04-25T08:59:16.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fe71518e-7b80-494f-b71d-fc3c2f3dadd1 -i td/osd-recovery-stats/5/new.json 2024-04-25T08:59:17.028 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:59:17.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T08:59:17.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBkGypmSJnTFxAAuQniML2LNLWPuUgeqEjMWQ== --osd-uuid fe71518e-7b80-494f-b71d-fc3c2f3dadd1 2024-04-25T08:59:17.082 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:17.092+0000 7f912739c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:17.084 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:17.092+0000 7f912739c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:17.086 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:17.096+0000 7f912739c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:17.086 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:17.096+0000 7f912739c7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:59:17.086 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:17.096+0000 7f912739c7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:59:17.086 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:17.096+0000 7f912739c7c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T08:59:17.158 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:17.168+0000 7f912739c7c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T08:59:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T08:59:24.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T08:59:24.005 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd5 key to auth repository 2024-04-25T08:59:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T08:59:24.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T08:59:24.550 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.5 2024-04-25T08:59:24.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T08:59:24.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T08:59:24.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T08:59:24.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T08:59:24.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T08:59:24.581 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:24.588+0000 7f431ec887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:24.592+0000 7f431ec887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:24.585 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:24.596+0000 7f431ec887c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T08:59:24.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T08:59:24.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T08:59:24.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T08:59:24.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T08:59:24.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T08:59:24.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:24.967 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T08:59:24.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T08:59:24.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:24.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:25.123 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:25.132+0000 7f431ec887c0 -1 Falling back to public interface 2024-04-25T08:59:25.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:26.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:26.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:26.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T08:59:26.377 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T08:59:26.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:26.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:26.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:27.800 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T08:59:27.808+0000 7f431ec887c0 -1 osd.5 0 log_to_monitors true 2024-04-25T08:59:27.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:27.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:27.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T08:59:27.825 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T08:59:27.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:27.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:28.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:29.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:29.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:29.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T08:59:29.598 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T08:59:29.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:29.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:30.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:31.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:31.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:31.060 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T08:59:31.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T08:59:31.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:31.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:31.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:32.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:32.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:32.532 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T08:59:32.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T08:59:32.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:32.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:33.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:34.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:34.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:34.077 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T08:59:34.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T08:59:34.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:34.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:34.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:35.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:35.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:35.519 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T08:59:35.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T08:59:35.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:35.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:36.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:37.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:37.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:37.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T08:59:37.089 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T08:59:37.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:37.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:37.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:38.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:38.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:38.532 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T08:59:38.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T08:59:38.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:38.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:39.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:40.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:40.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:40.041 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T08:59:40.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T08:59:40.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:40.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:40.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:41.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:41.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:41.478 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T08:59:41.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T08:59:41.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:41.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:42.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:43.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:43.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:43.266 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T08:59:43.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T08:59:43.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:43.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:44.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:45.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:45.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:45.372 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T08:59:45.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T08:59:45.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:45.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:46.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:47.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:47.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:47.244 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T08:59:47.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T08:59:47.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:47.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:47.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:48.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:48.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:48.773 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T08:59:48.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T08:59:48.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:48.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:49.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:50.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:50.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:50.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T08:59:50.939 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T08:59:50.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:50.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:51.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T08:59:52.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T08:59:52.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T08:59:52.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T08:59:52.858 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T08:59:52.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T08:59:52.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T08:59:53.320 INFO:tasks.workunit.client.0.smithi134.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/3446774684,v1:127.0.0.1:6843/3446774684] [v2:127.0.0.1:6844/3446774684,v1:127.0.0.1:6845/3446774684] exists,up fe71518e-7b80-494f-b71d-fc3c2f3dadd1 2024-04-25T08:59:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T08:59:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T08:59:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T08:59:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:252: TEST_recovery_sizedown: create_pool test 1 1 2024-04-25T08:59:53.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T08:59:53.997 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T08:59:54.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T08:59:55.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:253: TEST_recovery_sizedown: ceph osd pool set test size 4 2024-04-25T08:59:56.024 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 4 2024-04-25T08:59:56.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:255: TEST_recovery_sizedown: wait_for_clean 2024-04-25T08:59:56.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T08:59:56.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T08:59:56.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T08:59:56.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T08:59:56.048 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T08:59:56.048 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T08:59:56.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T08:59:56.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T08:59:56.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T08:59:56.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T08:59:56.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T08:59:56.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T08:59:56.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T08:59:56.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T08:59:56.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T08:59:56.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T08:59:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T08:59:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T08:59:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T08:59:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T08:59:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T08:59:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T08:59:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:59:56.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T08:59:56.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836526 2024-04-25T08:59:56.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836526 2024-04-25T08:59:56.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526' 2024-04-25T08:59:56.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:59:56.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T08:59:57.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=38654705703 2024-04-25T08:59:57.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 38654705703 2024-04-25T08:59:57.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-38654705703' 2024-04-25T08:59:57.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:59:57.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T08:59:57.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542172 2024-04-25T08:59:57.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542172 2024-04-25T08:59:57.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-38654705703 2-60129542172' 2024-04-25T08:59:57.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:59:57.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T08:59:57.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378646 2024-04-25T08:59:57.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378646 2024-04-25T08:59:57.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-38654705703 2-60129542172 3-81604378646' 2024-04-25T08:59:57.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:59:57.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T08:59:57.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215117 2024-04-25T08:59:57.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215117 2024-04-25T08:59:57.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-38654705703 2-60129542172 3-81604378646 4-103079215117' 2024-04-25T08:59:57.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T08:59:57.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T08:59:57.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051592 2024-04-25T08:59:57.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051592 2024-04-25T08:59:57.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836526 1-38654705703 2-60129542172 3-81604378646 4-103079215117 5-124554051592' 2024-04-25T08:59:57.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:59:57.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836526 2024-04-25T08:59:57.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:59:57.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T08:59:57.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836526 2024-04-25T08:59:57.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:59:57.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836526 2024-04-25T08:59:57.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2024-04-25T08:59:57.745 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836526 2024-04-25T08:59:57.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T08:59:58.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836526 -lt 21474836526 2024-04-25T08:59:58.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:59:58.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-38654705703 2024-04-25T08:59:58.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:59:58.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T08:59:58.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-38654705703 2024-04-25T08:59:58.184 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:59:58.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=38654705703 2024-04-25T08:59:58.185 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 38654705703 2024-04-25T08:59:58.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 38654705703' 2024-04-25T08:59:58.185 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T08:59:58.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705703 -lt 38654705703 2024-04-25T08:59:58.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:59:58.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542172 2024-04-25T08:59:58.625 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:59:58.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T08:59:58.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542172 2024-04-25T08:59:58.626 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:59:58.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542172 2024-04-25T08:59:58.628 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 60129542172 2024-04-25T08:59:58.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542172' 2024-04-25T08:59:58.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T08:59:59.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542172 -lt 60129542172 2024-04-25T08:59:59.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:59:59.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378646 2024-04-25T08:59:59.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:59:59.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T08:59:59.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:59:59.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378646 2024-04-25T08:59:59.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378646 2024-04-25T08:59:59.049 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378646 2024-04-25T08:59:59.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378646' 2024-04-25T08:59:59.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T08:59:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378646 -lt 81604378646 2024-04-25T08:59:59.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:59:59.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:59:59.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215117 2024-04-25T08:59:59.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T08:59:59.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215117 2024-04-25T08:59:59.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:59:59.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215117 2024-04-25T08:59:59.500 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215117 2024-04-25T08:59:59.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215117' 2024-04-25T08:59:59.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T08:59:59.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215117 -lt 103079215117 2024-04-25T08:59:59.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T08:59:59.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-124554051592 2024-04-25T08:59:59.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T08:59:59.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T08:59:59.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-124554051592 2024-04-25T08:59:59.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T08:59:59.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051592 2024-04-25T08:59:59.948 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 124554051592 2024-04-25T08:59:59.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 124554051592' 2024-04-25T08:59:59.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:00:00.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051593 -lt 124554051592 2024-04-25T09:00:00.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:00:00.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:00.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:01.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:00:01.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:00:01.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:00:01.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:00:01.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:00:01.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:00:01.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:00:01.001 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:00:01.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:00:01.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:00:01.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:01.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:01.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:00:01.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:00:01.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:00:01.966 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: seq 1 200 2024-04-25T09:00:01.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:01.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj1 /dev/null 2024-04-25T09:00:02.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:02.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj2 /dev/null 2024-04-25T09:00:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj3 /dev/null 2024-04-25T09:00:02.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:02.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj4 /dev/null 2024-04-25T09:00:02.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:02.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj5 /dev/null 2024-04-25T09:00:02.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:02.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj6 /dev/null 2024-04-25T09:00:03.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:03.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj7 /dev/null 2024-04-25T09:00:03.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:03.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj8 /dev/null 2024-04-25T09:00:03.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:03.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj9 /dev/null 2024-04-25T09:00:03.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:03.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj10 /dev/null 2024-04-25T09:00:03.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:03.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj11 /dev/null 2024-04-25T09:00:03.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:03.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj12 /dev/null 2024-04-25T09:00:04.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:04.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj13 /dev/null 2024-04-25T09:00:04.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:04.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj14 /dev/null 2024-04-25T09:00:04.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:04.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj15 /dev/null 2024-04-25T09:00:04.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:04.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj16 /dev/null 2024-04-25T09:00:04.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:04.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj17 /dev/null 2024-04-25T09:00:04.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:04.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj18 /dev/null 2024-04-25T09:00:05.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:05.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj19 /dev/null 2024-04-25T09:00:05.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:05.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj20 /dev/null 2024-04-25T09:00:05.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:05.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj21 /dev/null 2024-04-25T09:00:05.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:05.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj22 /dev/null 2024-04-25T09:00:05.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:05.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj23 /dev/null 2024-04-25T09:00:05.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:05.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj24 /dev/null 2024-04-25T09:00:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj25 /dev/null 2024-04-25T09:00:06.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:06.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj26 /dev/null 2024-04-25T09:00:06.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:06.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj27 /dev/null 2024-04-25T09:00:06.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:06.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj28 /dev/null 2024-04-25T09:00:06.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:06.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj29 /dev/null 2024-04-25T09:00:06.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:06.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj30 /dev/null 2024-04-25T09:00:07.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:07.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj31 /dev/null 2024-04-25T09:00:07.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:07.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj32 /dev/null 2024-04-25T09:00:07.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:07.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj33 /dev/null 2024-04-25T09:00:07.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:07.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj34 /dev/null 2024-04-25T09:00:07.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:07.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj35 /dev/null 2024-04-25T09:00:07.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:07.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj36 /dev/null 2024-04-25T09:00:07.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:07.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj37 /dev/null 2024-04-25T09:00:08.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:08.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj38 /dev/null 2024-04-25T09:00:08.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:08.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj39 /dev/null 2024-04-25T09:00:08.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:08.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj40 /dev/null 2024-04-25T09:00:08.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:08.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj41 /dev/null 2024-04-25T09:00:08.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:08.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj42 /dev/null 2024-04-25T09:00:08.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:08.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj43 /dev/null 2024-04-25T09:00:09.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:09.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj44 /dev/null 2024-04-25T09:00:09.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:09.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj45 /dev/null 2024-04-25T09:00:09.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:09.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj46 /dev/null 2024-04-25T09:00:09.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:09.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj47 /dev/null 2024-04-25T09:00:09.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:09.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj48 /dev/null 2024-04-25T09:00:09.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:09.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj49 /dev/null 2024-04-25T09:00:10.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:10.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj50 /dev/null 2024-04-25T09:00:10.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:10.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj51 /dev/null 2024-04-25T09:00:10.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:10.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj52 /dev/null 2024-04-25T09:00:10.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:10.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj53 /dev/null 2024-04-25T09:00:10.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:10.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj54 /dev/null 2024-04-25T09:00:10.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:10.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj55 /dev/null 2024-04-25T09:00:11.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:11.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj56 /dev/null 2024-04-25T09:00:11.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:11.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj57 /dev/null 2024-04-25T09:00:11.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:11.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj58 /dev/null 2024-04-25T09:00:11.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:11.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj59 /dev/null 2024-04-25T09:00:11.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:11.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj60 /dev/null 2024-04-25T09:00:11.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:11.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj61 /dev/null 2024-04-25T09:00:12.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:12.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj62 /dev/null 2024-04-25T09:00:12.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:12.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj63 /dev/null 2024-04-25T09:00:12.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:12.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj64 /dev/null 2024-04-25T09:00:12.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:12.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj65 /dev/null 2024-04-25T09:00:12.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:12.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj66 /dev/null 2024-04-25T09:00:12.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:12.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj67 /dev/null 2024-04-25T09:00:13.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:13.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj68 /dev/null 2024-04-25T09:00:13.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:13.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj69 /dev/null 2024-04-25T09:00:13.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:13.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj70 /dev/null 2024-04-25T09:00:13.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:13.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj71 /dev/null 2024-04-25T09:00:13.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:13.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj72 /dev/null 2024-04-25T09:00:13.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:13.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj73 /dev/null 2024-04-25T09:00:14.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:14.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj74 /dev/null 2024-04-25T09:00:14.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:14.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj75 /dev/null 2024-04-25T09:00:14.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:14.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj76 /dev/null 2024-04-25T09:00:14.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:14.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj77 /dev/null 2024-04-25T09:00:14.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:14.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj78 /dev/null 2024-04-25T09:00:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:14.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj79 /dev/null 2024-04-25T09:00:15.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:15.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj80 /dev/null 2024-04-25T09:00:15.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:15.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj81 /dev/null 2024-04-25T09:00:15.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:15.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj82 /dev/null 2024-04-25T09:00:15.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:15.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj83 /dev/null 2024-04-25T09:00:15.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:15.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj84 /dev/null 2024-04-25T09:00:15.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:15.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj85 /dev/null 2024-04-25T09:00:16.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:16.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj86 /dev/null 2024-04-25T09:00:16.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:16.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj87 /dev/null 2024-04-25T09:00:16.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:16.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj88 /dev/null 2024-04-25T09:00:16.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:16.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj89 /dev/null 2024-04-25T09:00:16.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:16.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj90 /dev/null 2024-04-25T09:00:16.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:16.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj91 /dev/null 2024-04-25T09:00:17.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:17.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj92 /dev/null 2024-04-25T09:00:17.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:17.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj93 /dev/null 2024-04-25T09:00:17.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:17.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj94 /dev/null 2024-04-25T09:00:17.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:17.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj95 /dev/null 2024-04-25T09:00:17.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:17.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj96 /dev/null 2024-04-25T09:00:17.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:17.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj97 /dev/null 2024-04-25T09:00:18.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:18.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj98 /dev/null 2024-04-25T09:00:18.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:18.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj99 /dev/null 2024-04-25T09:00:18.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:18.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj100 /dev/null 2024-04-25T09:00:18.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:18.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj101 /dev/null 2024-04-25T09:00:18.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:18.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj102 /dev/null 2024-04-25T09:00:18.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:18.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj103 /dev/null 2024-04-25T09:00:19.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:19.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj104 /dev/null 2024-04-25T09:00:19.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:19.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj105 /dev/null 2024-04-25T09:00:19.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:19.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj106 /dev/null 2024-04-25T09:00:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj107 /dev/null 2024-04-25T09:00:19.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:19.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj108 /dev/null 2024-04-25T09:00:19.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:19.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj109 /dev/null 2024-04-25T09:00:20.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:20.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj110 /dev/null 2024-04-25T09:00:20.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:20.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj111 /dev/null 2024-04-25T09:00:20.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:20.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj112 /dev/null 2024-04-25T09:00:20.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:20.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj113 /dev/null 2024-04-25T09:00:20.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:20.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj114 /dev/null 2024-04-25T09:00:20.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:20.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj115 /dev/null 2024-04-25T09:00:20.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:20.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj116 /dev/null 2024-04-25T09:00:21.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:21.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj117 /dev/null 2024-04-25T09:00:21.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:21.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj118 /dev/null 2024-04-25T09:00:21.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:21.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj119 /dev/null 2024-04-25T09:00:21.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:21.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj120 /dev/null 2024-04-25T09:00:21.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:21.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj121 /dev/null 2024-04-25T09:00:21.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:21.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj122 /dev/null 2024-04-25T09:00:22.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:22.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj123 /dev/null 2024-04-25T09:00:22.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:22.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj124 /dev/null 2024-04-25T09:00:22.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:22.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj125 /dev/null 2024-04-25T09:00:22.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:22.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj126 /dev/null 2024-04-25T09:00:22.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:22.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj127 /dev/null 2024-04-25T09:00:22.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:22.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj128 /dev/null 2024-04-25T09:00:23.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:23.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj129 /dev/null 2024-04-25T09:00:23.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:23.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj130 /dev/null 2024-04-25T09:00:23.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:23.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj131 /dev/null 2024-04-25T09:00:23.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:23.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj132 /dev/null 2024-04-25T09:00:23.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:23.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj133 /dev/null 2024-04-25T09:00:23.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:23.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj134 /dev/null 2024-04-25T09:00:24.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:24.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj135 /dev/null 2024-04-25T09:00:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:24.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj136 /dev/null 2024-04-25T09:00:24.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:24.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj137 /dev/null 2024-04-25T09:00:24.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:24.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj138 /dev/null 2024-04-25T09:00:24.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:24.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj139 /dev/null 2024-04-25T09:00:24.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:24.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj140 /dev/null 2024-04-25T09:00:25.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:25.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj141 /dev/null 2024-04-25T09:00:25.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:25.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj142 /dev/null 2024-04-25T09:00:25.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:25.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj143 /dev/null 2024-04-25T09:00:25.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:25.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj144 /dev/null 2024-04-25T09:00:25.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:25.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj145 /dev/null 2024-04-25T09:00:25.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:25.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj146 /dev/null 2024-04-25T09:00:26.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:26.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj147 /dev/null 2024-04-25T09:00:26.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:26.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj148 /dev/null 2024-04-25T09:00:26.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:26.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj149 /dev/null 2024-04-25T09:00:26.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:26.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj150 /dev/null 2024-04-25T09:00:26.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:26.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj151 /dev/null 2024-04-25T09:00:26.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:26.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj152 /dev/null 2024-04-25T09:00:27.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:27.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj153 /dev/null 2024-04-25T09:00:27.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:27.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj154 /dev/null 2024-04-25T09:00:27.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:27.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj155 /dev/null 2024-04-25T09:00:27.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:27.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj156 /dev/null 2024-04-25T09:00:27.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:27.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj157 /dev/null 2024-04-25T09:00:27.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:27.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj158 /dev/null 2024-04-25T09:00:28.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:28.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj159 /dev/null 2024-04-25T09:00:28.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:28.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj160 /dev/null 2024-04-25T09:00:28.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:28.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj161 /dev/null 2024-04-25T09:00:28.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:28.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj162 /dev/null 2024-04-25T09:00:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:28.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj163 /dev/null 2024-04-25T09:00:28.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:28.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj164 /dev/null 2024-04-25T09:00:29.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:29.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj165 /dev/null 2024-04-25T09:00:29.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:29.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj166 /dev/null 2024-04-25T09:00:29.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:29.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj167 /dev/null 2024-04-25T09:00:29.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:29.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj168 /dev/null 2024-04-25T09:00:29.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:29.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj169 /dev/null 2024-04-25T09:00:29.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:29.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj170 /dev/null 2024-04-25T09:00:29.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:29.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj171 /dev/null 2024-04-25T09:00:30.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:30.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj172 /dev/null 2024-04-25T09:00:30.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:30.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj173 /dev/null 2024-04-25T09:00:30.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:30.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj174 /dev/null 2024-04-25T09:00:30.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:30.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj175 /dev/null 2024-04-25T09:00:30.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:30.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj176 /dev/null 2024-04-25T09:00:30.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:30.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj177 /dev/null 2024-04-25T09:00:31.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:31.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj178 /dev/null 2024-04-25T09:00:31.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:31.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj179 /dev/null 2024-04-25T09:00:31.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:31.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj180 /dev/null 2024-04-25T09:00:31.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:31.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj181 /dev/null 2024-04-25T09:00:31.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:31.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj182 /dev/null 2024-04-25T09:00:31.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:31.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj183 /dev/null 2024-04-25T09:00:32.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:32.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj184 /dev/null 2024-04-25T09:00:32.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:32.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj185 /dev/null 2024-04-25T09:00:32.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:32.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj186 /dev/null 2024-04-25T09:00:32.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:32.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj187 /dev/null 2024-04-25T09:00:32.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:32.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj188 /dev/null 2024-04-25T09:00:32.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:32.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj189 /dev/null 2024-04-25T09:00:33.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:33.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj190 /dev/null 2024-04-25T09:00:33.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:33.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj191 /dev/null 2024-04-25T09:00:33.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:33.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj192 /dev/null 2024-04-25T09:00:33.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:33.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj193 /dev/null 2024-04-25T09:00:33.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:33.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj194 /dev/null 2024-04-25T09:00:33.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:33.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj195 /dev/null 2024-04-25T09:00:34.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:34.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj196 /dev/null 2024-04-25T09:00:34.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:34.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj197 /dev/null 2024-04-25T09:00:34.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:34.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj198 /dev/null 2024-04-25T09:00:34.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:34.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj199 /dev/null 2024-04-25T09:00:34.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:257: TEST_recovery_sizedown: for i in $(seq 1 $objects) 2024-04-25T09:00:34.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:259: TEST_recovery_sizedown: rados -p test put obj200 /dev/null 2024-04-25T09:00:34.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T09:00:34.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:00:34.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:00:34.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:00:34.905 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:00:35.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:262: TEST_recovery_sizedown: local primary=1 2024-04-25T09:00:35.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: get_pg test obj1 2024-04-25T09:00:35.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T09:00:35.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T09:00:35.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T09:00:35.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T09:00:35.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:263: TEST_recovery_sizedown: local PG=1.0 2024-04-25T09:00:35.782 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: get_osds test obj1 2024-04-25T09:00:35.782 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test 2024-04-25T09:00:35.782 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj1 2024-04-25T09:00:35.783 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test obj1 2024-04-25T09:00:35.783 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:00:36.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-04-25T09:00:36.248 INFO:tasks.workunit.client.0.smithi134.stderr:0 2024-04-25T09:00:36.248 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:00:36.248 INFO:tasks.workunit.client.0.smithi134.stderr:4' 2024-04-25T09:00:36.248 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2 4 2024-04-25T09:00:36.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:265: TEST_recovery_sizedown: local 'allosds=1 0 2 4' 2024-04-25T09:00:36.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:267: TEST_recovery_sizedown: ceph osd set norecover 2024-04-25T09:00:37.064 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is set 2024-04-25T09:00:37.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T09:00:37.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.1 2024-04-25T09:00:37.616 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already out. 2024-04-25T09:00:37.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T09:00:37.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.0 2024-04-25T09:00:39.112 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T09:00:39.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T09:00:39.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.2 2024-04-25T09:00:40.730 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already out. 2024-04-25T09:00:40.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:268: TEST_recovery_sizedown: for osd in $allosds 2024-04-25T09:00:40.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:270: TEST_recovery_sizedown: ceph osd out osd.4 2024-04-25T09:00:42.031 INFO:tasks.workunit.client.0.smithi134.stderr:osd.4 is already out. 2024-04-25T09:00:42.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:273: TEST_recovery_sizedown: ceph osd pool set test size 2 2024-04-25T09:00:43.369 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 2 2024-04-25T09:00:43.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:274: TEST_recovery_sizedown: ceph osd unset norecover 2024-04-25T09:00:44.610 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is unset 2024-04-25T09:00:44.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T09:00:44.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:00:44.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:00:44.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:00:44.634 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:00:45.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:275: TEST_recovery_sizedown: ceph tell osd.3 debug kick_recovery_wq 0 2024-04-25T09:00:45.404 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T09:00:45.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:276: TEST_recovery_sizedown: sleep 2 2024-04-25T09:00:47.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:278: TEST_recovery_sizedown: wait_for_clean 2024-04-25T09:00:47.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:00:47.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:00:47.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:00:47.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:00:47.421 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:00:47.421 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:00:47.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:00:47.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:00:47.422 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:00:47.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:00:47.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:00:47.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:00:47.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:00:47.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:00:47.548 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:00:47.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:00:47.973 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:00:47.973 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:00:47.973 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T09:00:47.973 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T09:00:47.973 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T09:00:47.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:00:47.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:47.974 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:00:48.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836538 2024-04-25T09:00:48.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836538 2024-04-25T09:00:48.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538' 2024-04-25T09:00:48.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:48.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:00:48.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=38654705715 2024-04-25T09:00:48.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 38654705715 2024-04-25T09:00:48.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-38654705715' 2024-04-25T09:00:48.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:48.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:00:48.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=60129542185 2024-04-25T09:00:48.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 60129542185 2024-04-25T09:00:48.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-38654705715 2-60129542185' 2024-04-25T09:00:48.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:48.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:00:48.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378658 2024-04-25T09:00:48.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378658 2024-04-25T09:00:48.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-38654705715 2-60129542185 3-81604378658' 2024-04-25T09:00:48.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:48.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:00:48.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=103079215129 2024-04-25T09:00:48.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 103079215129 2024-04-25T09:00:48.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-38654705715 2-60129542185 3-81604378658 4-103079215129' 2024-04-25T09:00:48.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:00:48.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:00:49.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051605 2024-04-25T09:00:49.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051605 2024-04-25T09:00:49.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836538 1-38654705715 2-60129542185 3-81604378658 4-103079215129 5-124554051605' 2024-04-25T09:00:49.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:49.027 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:49.027 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836538 2024-04-25T09:00:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:00:49.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836538 2024-04-25T09:00:49.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:49.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836538 2024-04-25T09:00:49.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836538' 2024-04-25T09:00:49.029 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836538 2024-04-25T09:00:49.029 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:00:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836536 -lt 21474836538 2024-04-25T09:00:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:00:50.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:00:50.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:00:50.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836539 -lt 21474836538 2024-04-25T09:00:50.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:50.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-38654705715 2024-04-25T09:00:50.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:50.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:00:50.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-38654705715 2024-04-25T09:00:50.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:50.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=38654705715 2024-04-25T09:00:50.891 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 38654705715 2024-04-25T09:00:50.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 38654705715' 2024-04-25T09:00:50.892 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:00:51.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 38654705715 -lt 38654705715 2024-04-25T09:00:51.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:51.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-60129542185 2024-04-25T09:00:51.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:51.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:00:51.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-60129542185 2024-04-25T09:00:51.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:51.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=60129542185 2024-04-25T09:00:51.369 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 60129542185 2024-04-25T09:00:51.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 60129542185' 2024-04-25T09:00:51.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:00:51.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 60129542185 -lt 60129542185 2024-04-25T09:00:51.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:51.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-81604378658 2024-04-25T09:00:51.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:51.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:00:51.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-81604378658 2024-04-25T09:00:51.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:51.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378658 2024-04-25T09:00:51.798 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 81604378658 2024-04-25T09:00:51.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 81604378658' 2024-04-25T09:00:51.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:00:52.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378658 -lt 81604378658 2024-04-25T09:00:52.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:52.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:52.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-103079215129 2024-04-25T09:00:52.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:00:52.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-103079215129 2024-04-25T09:00:52.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:52.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=103079215129 2024-04-25T09:00:52.302 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 103079215129 2024-04-25T09:00:52.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 103079215129' 2024-04-25T09:00:52.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:00:52.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 103079215130 -lt 103079215129 2024-04-25T09:00:52.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:00:52.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:00:52.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-124554051605 2024-04-25T09:00:52.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:00:52.883 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-124554051605 2024-04-25T09:00:52.883 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:00:52.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051605 2024-04-25T09:00:52.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 124554051605' 2024-04-25T09:00:52.884 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 124554051605 2024-04-25T09:00:52.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:00:53.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051605 -lt 124554051605 2024-04-25T09:00:53.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:00:53.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:53.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:00:53.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:00:53.925 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:00:53.925 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:00:53.925 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:00:53.925 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:00:53.925 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:00:53.925 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:00:54.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:00:54.327 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:00:54.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:54.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:54.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:00:54.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:00:54.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:00:54.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:00:54.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:00:54.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:00:55.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:00:55.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:00:55.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:00:55.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:00:55.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:00:55.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:00:55.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:00:55.100 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:00:55.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:00:55.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:00:55.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:55.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:56.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:00:56.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:00:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:00:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:00:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:00:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:00:56.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:00:56.220 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:00:56.220 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:00:56.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T09:00:56.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T09:00:56.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:00:56.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:00:56.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:00:56.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:00:56.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:00:56.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:00:56.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:00:56.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:00:56.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:00:56.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:00:56.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:00:57.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:00:57.321 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:00:57.321 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:57.321 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:57.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:00:57.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:00:57.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:00:57.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:00:57.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:00:57.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:00:57.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:00:57.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:00:57.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:00:58.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T09:00:58.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T09:00:58.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:00:58.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:00:58.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:00:58.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:00:58.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:00:58.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:00:58.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:00:58.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:00:58.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:00:58.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:00:58.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:00:59.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:00:59.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:00:59.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:00:59.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:00:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:00:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:00:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:00:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:00:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:00:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:00:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:00:59.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:00:59.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:00.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T09:01:00.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T09:01:00.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:00.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:00.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:00.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:00.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:00.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:00.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:00.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:00.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:00.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:00.319 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:00.734 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:00.735 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:00.735 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:01.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:01.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:01.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:01.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:01.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:01.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:01.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:01.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:01.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:01.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T09:01:01.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T09:01:01.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:01.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:01.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:01.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:01.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:01.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:01.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:01.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:01.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:01.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:01.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:02.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:02.336 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:02.336 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:02.336 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:02.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:02.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:02.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:02.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:02.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:02.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:02.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:02.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:02.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:03.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T09:01:03.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T09:01:03.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:03.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:03.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:03.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:03.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:03.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:03.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:03.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:03.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:03.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:03.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:04.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:04.160 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:04.160 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:04.160 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:04.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:04.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:04.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:04.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:04.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:04.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:04.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:04.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:04.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:05.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T09:01:05.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T09:01:05.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:05.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:05.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:05.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:05.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:05.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:05.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:05.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:05.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:05.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:05.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:05.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:05.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:05.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:05.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:06.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:06.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:06.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:06.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:06.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:06.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:06.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:06.426 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:06.426 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:06.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T09:01:06.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T09:01:06.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:06.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:06.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:07.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:07.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:07.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:07.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:07.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:07.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:07.087 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:07.087 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:07.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:07.604 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:07.604 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:07.604 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:08.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:08.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:08.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:08.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:08.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:08.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:08.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=8 2024-04-25T09:01:08.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 8 '!=' null 2024-04-25T09:01:08.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:08.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:08.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:08.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:08.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:08.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:08.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:08.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:08.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:08.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:08.896 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:09.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:09.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:09.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:09.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:10.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:10.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:10.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:10.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:10.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:10.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:10.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:10.208 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:10.208 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:10.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T09:01:10.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T09:01:10.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:10.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:10.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:10.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:10.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:10.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:10.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:10.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:10.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:10.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:10.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:11.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:11.381 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:11.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:11.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:11.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:11.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:11.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:11.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:11.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:11.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:11.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:11.907 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:11.907 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:12.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T09:01:12.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T09:01:12.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:12.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:12.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:12.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:12.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:12.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:12.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:12.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:12.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:12.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:12.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:13.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:13.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:13.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:13.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:13.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:13.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:13.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:13.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:13.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:13.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:13.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:13.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:13.616 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:14.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T09:01:14.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T09:01:14.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:14.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:14.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:14.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:14.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:14.554 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:14.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:14.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:14.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:14.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:14.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:14.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:14.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:14.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:14.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:15.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:15.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:15.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:15.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:15.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:15.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:15.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:15.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:15.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:16.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=7 2024-04-25T09:01:16.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 7 '!=' null 2024-04-25T09:01:16.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:16.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:16.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:16.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:16.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:16.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:16.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:16.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:16.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:16.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:16.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:16.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:01:16.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:16.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:16.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:17.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:01:17.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:01:17.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:01:17.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:01:17.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:01:17.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:01:17.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:01:17.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:01:17.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:01:17.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T09:01:17.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T09:01:17.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:01:17.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:01:17.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:01:18.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:01:18.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:01:18.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:01:18.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:01:18.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:01:18.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:01:18.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:01:18.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:01:18.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:01:18.407 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:01:18.407 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:01:18.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:01:19.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:01:19.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:01:19.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:01:19.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: get_primary test obj1 2024-04-25T09:01:19.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:01:19.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:01:19.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:01:19.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:01:19.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:281: TEST_recovery_sizedown: primary=3 2024-04-25T09:01:19.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: expr 200 '*' 2 2024-04-25T09:01:19.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:283: TEST_recovery_sizedown: local misplaced=400 2024-04-25T09:01:19.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:284: TEST_recovery_sizedown: local log=td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:285: TEST_recovery_sizedown: check td/osd-recovery-stats 1.0 3 replicated 0 0 400 0 2024-04-25T09:01:19.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T09:01:19.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T09:01:19.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=3 2024-04-25T09:01:19.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T09:01:19.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=0 2024-04-25T09:01:19.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T09:01:19.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-04-25T09:01:19.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T09:01:19.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T09:01:19.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T09:01:19.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T09:01:19.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T09:01:19.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T09:01:19.458 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.458 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T09:01:19.458 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T09:01:19.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[3,5]/[3]' 2024-04-25T09:01:19.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T09:01:19.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:01:19.477 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.477 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [3,5]/[3] ' 2024-04-25T09:01:19.477 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T09:01:19.477 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T09:01:19.478 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=0 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 0 0 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T09:01:19.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:19.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [3,5]/[3] ' 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T09:01:19.493 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:19.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-04-25T09:01:19.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-04-25T09:01:19.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T09:01:19.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:01:19.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T09:01:19.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:01:19.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:01:19.497 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.497 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T09:01:19.497 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:19.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T09:01:19.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T09:01:19.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T09:01:19.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:01:19.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T09:01:19.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:01:19.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T09:01:19.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats ' td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: tail -1 2024-04-25T09:01:19.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T09:01:19.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:287: TEST_recovery_sizedown: UPACT='[3,5]/[3]' 2024-04-25T09:01:19.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: grep -F ' [3,5]/[3] ' 2024-04-25T09:01:19.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: head -1 2024-04-25T09:01:19.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:290: TEST_recovery_sizedown: FIRST=200 2024-04-25T09:01:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:291: TEST_recovery_sizedown: below_margin 200 200 2024-04-25T09:01:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T09:01:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:01:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T09:01:19.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:01:19.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: grep 'pg[[]1.0.*recovering.*update_calc_stats shard 3 ' td/osd-recovery-stats/osd.3.log 2024-04-25T09:01:19.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: tail -1 2024-04-25T09:01:19.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:01:19.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:292: TEST_recovery_sizedown: LAST=0 2024-04-25T09:01:19.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:293: TEST_recovery_sizedown: above_margin 0 0 2024-04-25T09:01:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T09:01:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:01:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T09:01:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:01:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:295: TEST_recovery_sizedown: delete_pool test 2024-04-25T09:01:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T09:01:19.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T09:01:20.087 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T09:01:20.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:296: TEST_recovery_sizedown: kill_daemons td/osd-recovery-stats 2024-04-25T09:01:20.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:01:20.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:01:20.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:01:20.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:01:20.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:01:21.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:01:21.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T09:01:21.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T09:01:21.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:01:21.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T09:01:21.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:01:21.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:01:21.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:01:21.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:01:21.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:01:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:01:21.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:01:21.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:01:21.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:01:21.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:01:21.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:01:21.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:01:21.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:21.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:01:21.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:01:21.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:21.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:01:21.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:01:21.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T09:01:21.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:01:21.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:21.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:01:21.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T09:01:21.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:01:21.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:01:21.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T09:01:21.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T09:01:21.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T09:01:21.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T09:01:21.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T09:01:21.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:01:21.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T09:01:21.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:01:21.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:01:21.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:01:21.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:01:21.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:01:21.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:01:21.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:01:21.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:01:21.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:01:21.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:01:21.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:01:21.485 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:01:21.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:21.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:01:21.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:01:21.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:01:21.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:01:21.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:01:21.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T09:01:21.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:01:21.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:21.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:01:21.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T09:01:21.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:01:21.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:01:21.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T09:01:21.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:01:21.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:21.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:01:21.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.280166 2024-04-25T09:01:21.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_sizeup td/osd-recovery-stats 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:186: TEST_recovery_sizeup: local dir=td/osd-recovery-stats 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:188: TEST_recovery_sizeup: run_mon td/osd-recovery-stats a 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T09:01:21.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T09:01:21.973 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:01:21.974 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:21.974 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:21.974 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:21.975 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:21.975 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:01:21.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:01:21.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:01:22.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:01:22.212 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:01:22.212 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:01:22.212 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:01:22.212 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:01:22.213 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:01:22.213 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:01:22.213 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:01:22.213 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:01:22.213 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:01:22.213 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:22.213 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:01:22.213 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T09:01:22.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:01:22.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get fsid 2024-04-25T09:01:22.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:01:22.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:01:22.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:01:22.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:01:22.324 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:01:22.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:01:22.324 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:01:22.324 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:01:22.324 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:01:22.324 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:22.324 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:01:22.325 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T09:01:22.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:01:22.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get mon_host 2024-04-25T09:01:22.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:189: TEST_recovery_sizeup: run_mgr td/osd-recovery-stats x 2024-04-25T09:01:22.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T09:01:22.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:01:22.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:01:22.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:01:22.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T09:01:22.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:01:23.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:01:23.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:23.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:23.341 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:23.342 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:23.342 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:01:23.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:01:23.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:01:23.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:01:23.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:190: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 0 2024-04-25T09:01:23.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:01:23.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:01:23.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:01:23.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:01:23.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:01:23.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:01:23.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:01:23.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:01:23.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:01:23.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:01:23.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T09:01:23.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:01:23.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7b067ecc-c336-4096-a7bc-925941006b99 2024-04-25T09:01:23.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7b067ecc-c336-4096-a7bc-925941006b99' 2024-04-25T09:01:23.380 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 7b067ecc-c336-4096-a7bc-925941006b99 2024-04-25T09:01:23.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:01:23.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDjGypmDps0GBAAiSj9FUA+11F9yCGw3ZVA+g== 2024-04-25T09:01:23.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDjGypmDps0GBAAiSj9FUA+11F9yCGw3ZVA+g=="}' 2024-04-25T09:01:23.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7b067ecc-c336-4096-a7bc-925941006b99 -i td/osd-recovery-stats/0/new.json 2024-04-25T09:01:23.721 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:01:23.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T09:01:23.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDjGypmDps0GBAAiSj9FUA+11F9yCGw3ZVA+g== --osd-uuid 7b067ecc-c336-4096-a7bc-925941006b99 2024-04-25T09:01:23.777 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:23.789+0000 7f8e56d5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:23.798 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:23.809+0000 7f8e56d5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:23.799 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:23.809+0000 7f8e56d5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:23.800 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:23.809+0000 7f8e56d5c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T09:01:23.800 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:23.809+0000 7f8e56d5c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T09:01:23.800 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:23.809+0000 7f8e56d5c7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T09:01:23.838 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:23.849+0000 7f8e56d5c7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T09:01:30.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T09:01:30.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:01:30.097 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:01:30.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:01:30.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:01:30.640 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:01:30.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:01:30.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:01:30.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:01:30.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:01:30.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:01:30.672 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:30.681+0000 7f00aa3a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:30.694 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:30.705+0000 7f00aa3a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:30.696 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:30.705+0000 7f00aa3a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:01:31.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:01:31.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:01:31.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:01:31.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:01:31.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:01:31.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:31.072 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:01:31.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:01:31.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:31.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:31.257+0000 7f00aa3a67c0 -1 Falling back to public interface 2024-04-25T09:01:31.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:32.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:01:32.505 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:01:32.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:32.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:32.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:33.876 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:33.885+0000 7f00aa3a67c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:01:33.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:33.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:33.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:01:33.917 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:01:33.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:33.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:34.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:35.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:35.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:01:35.609 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:01:35.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:35.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:36.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:37.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:37.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:37.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:01:37.080 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:01:37.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:37.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:37.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:38.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:38.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:38.539 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:01:38.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:01:38.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:38.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:39.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:40.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:40.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:40.085 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:01:40.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:01:40.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:40.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:40.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:41.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:41.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:41.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:01:41.547 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:01:41.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:41.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:42.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:43.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:43.047 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:01:43.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:43.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:01:43.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:43.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:43.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:44.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:44.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:44.480 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:01:44.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:01:44.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:44.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:45.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:46.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:46.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:46.080 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:01:46.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:01:46.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:46.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:46.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:47.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:47.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:47.518 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:01:47.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:01:47.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:47.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:48.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:49.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:49.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:49.414 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:01:49.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:01:49.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:49.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:50.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:51.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:51.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:51.610 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:01:51.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:01:51.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:51.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:52.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:53.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:53.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:53.432 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:01:53.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:01:53.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:53.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:54.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:55.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:55.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:55.206 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:01:55.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:01:55.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:55.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:55.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:01:56.935 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:01:56.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:56.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:57.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:01:58.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:01:58.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:01:58.828 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:01:58.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:01:58.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:01:58.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:01:59.428 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4018476451,v1:127.0.0.1:6803/4018476451] [v2:127.0.0.1:6804/4018476451,v1:127.0.0.1:6805/4018476451] exists,up 7b067ecc-c336-4096-a7bc-925941006b99 2024-04-25T09:01:59.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:01:59.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:01:59.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:01:59.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:191: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 1 2024-04-25T09:01:59.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:01:59.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:01:59.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:01:59.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:01:59.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T09:01:59.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:01:59.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:01:59.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:01:59.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:01:59.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T09:01:59.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T09:01:59.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:01:59.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:01:59.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T09:01:59.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:01:59.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ebae7698-a34b-42cd-8e3d-2fe7a7e87a02 2024-04-25T09:01:59.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 ebae7698-a34b-42cd-8e3d-2fe7a7e87a02' 2024-04-25T09:01:59.437 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 ebae7698-a34b-42cd-8e3d-2fe7a7e87a02 2024-04-25T09:01:59.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:01:59.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHHCpmrDHWGxAAv+5KGOPe3o9Lha1Vq5afPg== 2024-04-25T09:01:59.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHHCpmrDHWGxAAv+5KGOPe3o9Lha1Vq5afPg=="}' 2024-04-25T09:01:59.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ebae7698-a34b-42cd-8e3d-2fe7a7e87a02 -i td/osd-recovery-stats/1/new.json 2024-04-25T09:01:59.926 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:01:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T09:01:59.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHHCpmrDHWGxAAv+5KGOPe3o9Lha1Vq5afPg== --osd-uuid ebae7698-a34b-42cd-8e3d-2fe7a7e87a02 2024-04-25T09:01:59.981 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:01:59.992+0000 7fe903eb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:00.019 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:00.028+0000 7fe903eb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:00.020 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:00.028+0000 7fe903eb17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:00.021 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:00.028+0000 7fe903eb17c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T09:02:00.021 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:00.032+0000 7fe903eb17c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T09:02:00.021 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:00.032+0000 7fe903eb17c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T09:02:00.143 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:00.152+0000 7fe903eb17c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T09:02:06.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T09:02:06.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:02:06.792 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:02:06.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:02:06.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:02:07.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:02:07.358 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:02:07.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:02:07.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:02:07.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:02:07.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:02:07.385 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:07.396+0000 7f52ec9357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:07.387 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:07.396+0000 7f52ec9357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:07.388 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:07.396+0000 7f52ec9357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:07.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:02:07.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:02:07.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:02:07.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:02:07.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:02:07.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:07.830 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:02:07.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:02:07.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:07.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:07.958 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:07.968+0000 7f52ec9357c0 -1 Falling back to public interface 2024-04-25T09:02:08.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:09.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:09.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:09.253 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:02:09.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:02:09.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:09.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:09.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:10.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:10.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:10.674 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:02:10.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:02:10.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:10.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:10.695 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:10.704+0000 7f52ec9357c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:02:11.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:12.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:12.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:12.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:02:12.254 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:02:12.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:12.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:12.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:13.730 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:02:13.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:13.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:13.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:02:13.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:13.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:14.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:15.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:15.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:15.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:02:15.178 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:02:15.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:15.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:15.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:16.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:16.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:16.749 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:02:16.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:02:16.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:16.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:17.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:18.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:18.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:18.213 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:02:18.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:02:18.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:18.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:18.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:19.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:19.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:19.756 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:02:19.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:02:19.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:19.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:20.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:21.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:21.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:21.176 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:02:21.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:02:21.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:21.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:21.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:22.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:22.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:22.731 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:02:22.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:02:22.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:22.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:23.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:24.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:24.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:24.355 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:02:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:02:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:24.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:25.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:26.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:26.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:26.425 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:02:26.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:02:26.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:26.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:27.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:28.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:28.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:28.093 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:02:28.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:02:28.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:28.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:28.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:29.749 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:02:29.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:02:29.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:29.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:30.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:31.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:31.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:31.599 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:02:31.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:02:31.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:31.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:32.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:33.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:33.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:33.439 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:02:33.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:02:33.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:33.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:34.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:35.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:35.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:35.213 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:02:35.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:02:35.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:35.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:35.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:36.846 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:02:36.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:02:36.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:36.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:02:37.268 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/4174828917,v1:127.0.0.1:6811/4174828917] [v2:127.0.0.1:6812/4174828917,v1:127.0.0.1:6813/4174828917] exists,up ebae7698-a34b-42cd-8e3d-2fe7a7e87a02 2024-04-25T09:02:37.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:02:37.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:02:37.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:02:37.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:192: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 2 2024-04-25T09:02:37.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:02:37.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:02:37.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:02:37.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:02:37.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:02:37.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:02:37.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:02:37.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:02:37.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:02:37.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:02:37.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:02:37.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T09:02:37.276 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:02:37.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c2759ae3-7567-4c8e-bb12-2985f47b9092 2024-04-25T09:02:37.276 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 c2759ae3-7567-4c8e-bb12-2985f47b9092 2024-04-25T09:02:37.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 c2759ae3-7567-4c8e-bb12-2985f47b9092' 2024-04-25T09:02:37.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:02:37.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtHCpmzfAVEhAA+YSEhvwG2r4cnCZozm76Iw== 2024-04-25T09:02:37.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtHCpmzfAVEhAA+YSEhvwG2r4cnCZozm76Iw=="}' 2024-04-25T09:02:37.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c2759ae3-7567-4c8e-bb12-2985f47b9092 -i td/osd-recovery-stats/2/new.json 2024-04-25T09:02:37.780 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:02:37.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T09:02:37.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtHCpmzfAVEhAA+YSEhvwG2r4cnCZozm76Iw== --osd-uuid c2759ae3-7567-4c8e-bb12-2985f47b9092 2024-04-25T09:02:37.826 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:37.835+0000 7f571a2de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:37.832 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:37.839+0000 7f571a2de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:37.833 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:37.843+0000 7f571a2de7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:37.833 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:37.843+0000 7f571a2de7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T09:02:37.833 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:37.843+0000 7f571a2de7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T09:02:37.833 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:37.843+0000 7f571a2de7c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T09:02:37.871 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:37.879+0000 7f571a2de7c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T09:02:44.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T09:02:44.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:02:44.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:02:44.594 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:02:44.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:02:45.146 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:02:45.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:02:45.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:02:45.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:02:45.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:02:45.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:02:45.179 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:45.187+0000 7f0a283b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:45.181 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:45.191+0000 7f0a283b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:45.182 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:45.191+0000 7f0a283b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:02:45.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:02:45.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:02:45.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:02:45.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:02:45.621 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:02:45.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:02:45.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:45.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:02:45.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:45.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:45.746 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:45.755+0000 7f0a283b47c0 -1 Falling back to public interface 2024-04-25T09:02:46.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:47.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:47.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:47.041 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:02:47.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:02:47.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:47.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:47.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:48.350 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:02:48.359+0000 7f0a283b47c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:02:48.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:48.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:48.448 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:02:48.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:02:48.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:48.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:49.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:50.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:50.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:50.113 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:02:50.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:02:50.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:50.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:50.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:51.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:51.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:51.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:02:51.702 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:02:51.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:51.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:52.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:53.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:53.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:53.276 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:02:53.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:02:53.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:53.277 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:53.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:54.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:54.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:54.698 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:02:54.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:02:54.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:54.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:55.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:56.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:56.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:56.213 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:02:56.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:02:56.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:56.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:56.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:57.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:57.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:57.695 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:02:57.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:02:57.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:57.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:58.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:02:59.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:02:59.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:02:59.312 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:02:59.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:02:59.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:02:59.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:02:59.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:00.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:00.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:00.786 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:03:00.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:03:00.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:00.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:01.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:02.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:02.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:02.334 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:03:02.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:03:02.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:02.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:02.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:03.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:03.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:03.819 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:03:03.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:03:03.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:03.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:04.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:05.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:05.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:05.822 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:03:05.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:03:05.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:05.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:06.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:07.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:07.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:07.741 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:03:07.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:03:07.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:07.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:08.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:09.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:09.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:09.583 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:03:09.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:03:09.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:09.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:10.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:11.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:11.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:11.379 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:03:11.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:03:11.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:11.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:12.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:13.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:13.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:13.504 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:03:13.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:03:13.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:13.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:03:13.982 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1485382669,v1:127.0.0.1:6819/1485382669] [v2:127.0.0.1:6820/1485382669,v1:127.0.0.1:6821/1485382669] exists,up c2759ae3-7567-4c8e-bb12-2985f47b9092 2024-04-25T09:03:13.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:03:13.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:03:13.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:193: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 3 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/3 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/3' 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/3/journal' 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:03:13.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:03:13.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:03:13.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:03:13.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:03:13.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:03:13.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:03:13.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/3 2024-04-25T09:03:13.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:03:13.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=26e02e7f-39a0-40f6-944b-758fb6f81d32 2024-04-25T09:03:13.990 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 26e02e7f-39a0-40f6-944b-758fb6f81d32 2024-04-25T09:03:13.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 26e02e7f-39a0-40f6-944b-758fb6f81d32' 2024-04-25T09:03:13.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:03:14.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSHCpm8LT7ABAA9qFKGMwqyVzrHa9RUbw6eQ== 2024-04-25T09:03:14.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSHCpm8LT7ABAA9qFKGMwqyVzrHa9RUbw6eQ=="}' 2024-04-25T09:03:14.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 26e02e7f-39a0-40f6-944b-758fb6f81d32 -i td/osd-recovery-stats/3/new.json 2024-04-25T09:03:14.646 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:03:14.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/3/new.json 2024-04-25T09:03:14.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSHCpm8LT7ABAA9qFKGMwqyVzrHa9RUbw6eQ== --osd-uuid 26e02e7f-39a0-40f6-944b-758fb6f81d32 2024-04-25T09:03:14.694 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:14.702+0000 7fe25afc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:14.728 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:14.734+0000 7fe25afc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:14.729 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:14.738+0000 7fe25afc47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:14.730 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:14.738+0000 7fe25afc47c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T09:03:14.730 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:14.738+0000 7fe25afc47c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T09:03:14.730 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:14.738+0000 7fe25afc47c0 -1 bluestore(td/osd-recovery-stats/3/block) _read_bdev_label failed to open td/osd-recovery-stats/3/block: (2) No such file or directory 2024-04-25T09:03:14.835 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:14.842+0000 7fe25afc47c0 -1 bluestore(td/osd-recovery-stats/3) _read_fsid unparsable uuid 2024-04-25T09:03:21.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/3/keyring 2024-04-25T09:03:21.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:03:21.563 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T09:03:21.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:03:21.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:03:22.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:03:22.136 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T09:03:22.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/3 --osd-journal=td/osd-recovery-stats/3/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:03:22.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:03:22.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:03:22.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:03:22.161 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:22.170+0000 7fbad84757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:22.163 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:22.170+0000 7fbad84757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:22.165 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:22.174+0000 7fbad84757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:03:22.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:03:22.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:03:22.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:03:22.600 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:03:22.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:03:22.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:22.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:03:22.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:22.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:22.734 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:22.742+0000 7fbad84757c0 -1 Falling back to public interface 2024-04-25T09:03:23.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:24.049 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:03:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:03:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:24.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:25.349 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:25.358+0000 7fbad84757c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:03:25.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:25.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:25.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:03:25.466 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:03:25.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:25.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:26.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:27.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:27.264 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:03:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:03:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:27.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:27.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:28.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:28.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:28.745 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:03:28.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:03:28.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:28.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:29.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:30.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:30.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:30.311 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:03:30.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:03:30.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:30.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:30.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:31.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:31.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:31.880 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:03:31.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:03:31.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:31.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:32.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:33.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:33.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:33.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:03:33.370 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:03:33.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:33.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:33.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:34.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:34.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:34.841 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:03:34.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:03:34.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:34.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:35.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:36.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:36.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:36.340 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:03:36.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:03:36.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:36.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:36.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:37.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:37.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:37.895 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:03:37.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:03:37.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:37.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:38.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:39.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:39.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:39.351 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:03:39.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:03:39.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:39.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:39.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:40.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:40.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:40.786 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:03:40.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:03:40.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:40.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:41.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:42.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:42.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:42.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:03:42.703 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:03:42.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:42.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:43.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:44.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:44.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:44.511 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:03:44.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:03:44.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:44.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:45.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:46.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:46.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:46.350 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:03:46.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:03:46.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:46.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:47.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:48.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:48.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:48.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:03:48.158 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:03:48.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:48.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:48.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:49.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:49.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:49.970 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:03:49.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:03:49.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:49.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:50.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:03:51.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:03:51.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:03:51.470 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:03:51.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:03:51.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:03:51.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:03:51.983 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3899658512,v1:127.0.0.1:6827/3899658512] [v2:127.0.0.1:6828/3899658512,v1:127.0.0.1:6829/3899658512] exists,up 26e02e7f-39a0-40f6-944b-758fb6f81d32 2024-04-25T09:03:51.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:03:51.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:194: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 4 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/4 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/4' 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/4/journal' 2024-04-25T09:03:51.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:03:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:03:51.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:03:51.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:03:51.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:03:51.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:03:51.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:03:51.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:03:51.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:03:51.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:03:51.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/4 2024-04-25T09:03:51.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:03:51.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f41748e-f37f-44e2-b61e-310ab2510647 2024-04-25T09:03:51.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 9f41748e-f37f-44e2-b61e-310ab2510647' 2024-04-25T09:03:51.990 INFO:tasks.workunit.client.0.smithi134.stdout:add osd4 9f41748e-f37f-44e2-b61e-310ab2510647 2024-04-25T09:03:51.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:03:52.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB4HCpmqk1gARAAnmF9Xsw+x4SlUSEK8aSSVg== 2024-04-25T09:03:52.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB4HCpmqk1gARAAnmF9Xsw+x4SlUSEK8aSSVg=="}' 2024-04-25T09:03:52.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f41748e-f37f-44e2-b61e-310ab2510647 -i td/osd-recovery-stats/4/new.json 2024-04-25T09:03:52.626 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:03:52.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/4/new.json 2024-04-25T09:03:52.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB4HCpmqk1gARAAnmF9Xsw+x4SlUSEK8aSSVg== --osd-uuid 9f41748e-f37f-44e2-b61e-310ab2510647 2024-04-25T09:03:52.672 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:52.681+0000 7fd3ff79d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:52.674 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:52.685+0000 7fd3ff79d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:52.675 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:52.685+0000 7fd3ff79d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:52.676 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:52.685+0000 7fd3ff79d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T09:03:52.676 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:52.685+0000 7fd3ff79d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T09:03:52.676 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:52.685+0000 7fd3ff79d7c0 -1 bluestore(td/osd-recovery-stats/4/block) _read_bdev_label failed to open td/osd-recovery-stats/4/block: (2) No such file or directory 2024-04-25T09:03:52.807 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:52.817+0000 7fd3ff79d7c0 -1 bluestore(td/osd-recovery-stats/4) _read_fsid unparsable uuid 2024-04-25T09:03:59.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/4/keyring 2024-04-25T09:03:59.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:03:59.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T09:03:59.365 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd4 key to auth repository 2024-04-25T09:03:59.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:03:59.933 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.4 2024-04-25T09:03:59.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T09:03:59.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/4 --osd-journal=td/osd-recovery-stats/4/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:03:59.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:03:59.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:03:59.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:03:59.964 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:59.973+0000 7f8f7128e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:59.966 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:59.977+0000 7f8f7128e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:03:59.968 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:03:59.977+0000 7f8f7128e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:00.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T09:04:00.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:04:00.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T09:04:00.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:04:00.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:04:00.366 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:04:00.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:00.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:04:00.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:00.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:00.534 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:00.545+0000 7f8f7128e7c0 -1 Falling back to public interface 2024-04-25T09:04:00.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:01.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:01.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:01.890 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:04:01.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:04:01.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:01.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:02.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:03.170 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:03.177+0000 7f8f7128e7c0 -1 osd.4 0 log_to_monitors true 2024-04-25T09:04:03.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:03.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:03.313 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:04:03.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:04:03.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:03.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:03.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:04.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:04.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:04.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:04:04.917 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:04:04.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:04.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:05.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:06.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:06.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:06.454 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:04:06.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:04:06.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:06.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:06.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:07.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:07.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:07.879 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:04:07.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:04:07.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:07.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:08.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:09.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:09.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:09.487 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:04:09.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:04:09.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:09.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:10.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:11.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:11.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:11.048 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:04:11.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:04:11.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:11.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:11.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:12.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:12.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:12.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:04:12.546 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:04:12.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:12.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:13.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:14.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:14.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:14.125 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:04:14.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:04:14.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:14.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:14.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:15.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:15.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:15.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:04:15.605 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:04:15.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:15.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:16.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:17.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:17.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:17.184 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:04:17.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:04:17.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:17.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:17.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:18.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:18.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:18.891 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:04:18.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:04:18.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:18.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:19.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:20.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:20.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:20.450 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:04:20.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:04:20.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:20.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:21.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:22.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:22.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:22.568 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:04:22.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:04:22.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:22.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:23.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:24.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:24.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:24.712 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:04:24.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:04:24.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:24.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:25.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:26.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:26.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:26.944 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:04:26.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:04:26.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:26.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:27.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:28.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:28.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:28.792 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:04:28.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:04:28.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:28.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:04:29.299 INFO:tasks.workunit.client.0.smithi134.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1981064190,v1:127.0.0.1:6835/1981064190] [v2:127.0.0.1:6836/1981064190,v1:127.0.0.1:6837/1981064190] exists,up 9f41748e-f37f-44e2-b61e-310ab2510647 2024-04-25T09:04:29.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:04:29.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:04:29.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:195: TEST_recovery_sizeup: run_osd td/osd-recovery-stats 5 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/5 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/5' 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/5/journal' 2024-04-25T09:04:29.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:04:29.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:04:29.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:04:29.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:04:29.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:04:29.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:04:29.303 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:04:29.303 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:04:29.303 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:04:29.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:04:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:04:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:04:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:04:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:04:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:04:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:04:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:04:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:04:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:04:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:04:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:04:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:04:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/5 2024-04-25T09:04:29.305 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:04:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=37bed585-4dec-4887-9774-67e16cdd8151 2024-04-25T09:04:29.305 INFO:tasks.workunit.client.0.smithi134.stdout:add osd5 37bed585-4dec-4887-9774-67e16cdd8151 2024-04-25T09:04:29.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 37bed585-4dec-4887-9774-67e16cdd8151' 2024-04-25T09:04:29.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:04:29.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdHCpmLt74ExAA1xq31xtsS4sfXWq+uXOONw== 2024-04-25T09:04:29.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdHCpmLt74ExAA1xq31xtsS4sfXWq+uXOONw=="}' 2024-04-25T09:04:29.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 37bed585-4dec-4887-9774-67e16cdd8151 -i td/osd-recovery-stats/5/new.json 2024-04-25T09:04:29.898 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:04:29.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/5/new.json 2024-04-25T09:04:29.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdHCpmLt74ExAA1xq31xtsS4sfXWq+uXOONw== --osd-uuid 37bed585-4dec-4887-9774-67e16cdd8151 2024-04-25T09:04:29.948 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:29.957+0000 7f5b7f1727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:30.012 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:30.021+0000 7f5b7f1727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:30.014 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:30.025+0000 7f5b7f1727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:30.014 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:30.025+0000 7f5b7f1727c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T09:04:30.014 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:30.025+0000 7f5b7f1727c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T09:04:30.014 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:30.025+0000 7f5b7f1727c0 -1 bluestore(td/osd-recovery-stats/5/block) _read_bdev_label failed to open td/osd-recovery-stats/5/block: (2) No such file or directory 2024-04-25T09:04:30.128 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:30.137+0000 7f5b7f1727c0 -1 bluestore(td/osd-recovery-stats/5) _read_fsid unparsable uuid 2024-04-25T09:04:37.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/5/keyring 2024-04-25T09:04:37.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:04:37.471 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd5 key to auth repository 2024-04-25T09:04:37.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T09:04:37.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:04:38.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T09:04:38.039 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.5 2024-04-25T09:04:38.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:04:38.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:04:38.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:04:38.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/5 --osd-journal=td/osd-recovery-stats/5/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:04:38.073 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:38.084+0000 7f559edca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:38.089 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:38.100+0000 7f559edca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:38.090 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:38.100+0000 7f559edca7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:04:38.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T09:04:38.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:04:38.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T09:04:38.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:04:38.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:04:38.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:38.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:04:38.457 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:04:38.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:38.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:38.667 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:38.676+0000 7f559edca7c0 -1 Falling back to public interface 2024-04-25T09:04:38.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:39.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:39.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:39.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:04:39.903 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:04:39.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:39.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:40.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:41.256 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:04:41.264+0000 7f559edca7c0 -1 osd.5 0 log_to_monitors true 2024-04-25T09:04:41.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:41.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:41.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:04:41.314 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:04:41.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:41.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:41.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:42.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:42.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:42.940 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:04:42.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:04:42.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:42.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:43.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:44.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:44.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:44.545 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:04:44.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:04:44.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:44.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:45.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:46.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:46.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:46.005 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:04:46.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:04:46.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:46.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:46.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:47.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:47.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:47.554 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:04:47.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:04:47.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:47.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:48.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:49.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:49.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:49.002 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:04:49.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:04:49.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:49.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:49.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:50.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:50.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:50.535 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:04:50.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:04:50.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:50.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:51.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:52.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:52.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:52.102 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:04:52.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:04:52.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:52.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:52.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:53.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:53.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:53.581 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:04:53.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:04:53.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:53.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:54.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:55.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:55.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:55.020 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:04:55.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:04:55.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:55.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:55.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:56.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:56.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:56.596 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:04:56.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:04:56.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:56.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:57.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:04:58.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:04:58.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:04:58.637 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:04:58.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:04:58.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:04:58.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:04:59.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:00.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:00.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:00.415 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:05:00.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:05:00.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:00.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:05:01.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:02.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:02.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:02.668 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:05:02.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:05:02.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:02.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:05:03.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:04.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:04.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:04.551 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:05:04.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:05:04.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:04.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:05:05.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:06.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:06.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:06.148 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:05:06.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:05:06.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:06.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:05:07.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:05:08.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:05:08.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:05:08.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:05:08.236 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:05:08.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:05:08.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:05:08.667 INFO:tasks.workunit.client.0.smithi134.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2533980484,v1:127.0.0.1:6843/2533980484] [v2:127.0.0.1:6844/2533980484,v1:127.0.0.1:6845/2533980484] exists,up 37bed585-4dec-4887-9774-67e16cdd8151 2024-04-25T09:05:08.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:05:08.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:05:08.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:05:08.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:197: TEST_recovery_sizeup: create_pool test 1 1 2024-04-25T09:05:08.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T09:05:09.354 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T09:05:09.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:05:10.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:198: TEST_recovery_sizeup: ceph osd pool set test size 2 2024-04-25T09:05:11.470 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 2 2024-04-25T09:05:11.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:200: TEST_recovery_sizeup: wait_for_clean 2024-04-25T09:05:11.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:05:11.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:05:11.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:05:11.491 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:05:11.492 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:05:11.492 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:05:11.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:05:11.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:05:11.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:05:11.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:05:11.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:05:11.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:05:11.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:05:11.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:05:11.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:05:12.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:05:12.394 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:05:12.394 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:05:12.394 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T09:05:12.394 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T09:05:12.394 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T09:05:12.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:05:12.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:12.394 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:05:12.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836527 2024-04-25T09:05:12.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836527 2024-04-25T09:05:12.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527' 2024-04-25T09:05:12.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:12.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:05:12.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672995 2024-04-25T09:05:12.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672995 2024-04-25T09:05:12.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672995' 2024-04-25T09:05:12.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:12.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:05:12.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509472 2024-04-25T09:05:12.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509472 2024-04-25T09:05:12.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672995 2-64424509472' 2024-04-25T09:05:12.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:12.893 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:05:13.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345944 2024-04-25T09:05:13.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345944 2024-04-25T09:05:13.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672995 2-64424509472 3-85899345944' 2024-04-25T09:05:13.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:13.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:05:13.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182416 2024-04-25T09:05:13.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182416 2024-04-25T09:05:13.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672995 2-64424509472 3-85899345944 4-107374182416' 2024-04-25T09:05:13.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:13.211 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:05:13.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018886 2024-04-25T09:05:13.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018886 2024-04-25T09:05:13.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836527 1-42949672995 2-64424509472 3-85899345944 4-107374182416 5-128849018886' 2024-04-25T09:05:13.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:13.361 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836527 2024-04-25T09:05:13.361 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:13.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:05:13.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836527 2024-04-25T09:05:13.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:13.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836527 2024-04-25T09:05:13.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836527' 2024-04-25T09:05:13.364 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836527 2024-04-25T09:05:13.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:13.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836525 -lt 21474836527 2024-04-25T09:05:13.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:05:14.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:05:14.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:15.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836527 -lt 21474836527 2024-04-25T09:05:15.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:15.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672995 2024-04-25T09:05:15.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:15.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:05:15.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672995 2024-04-25T09:05:15.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:15.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672995 2024-04-25T09:05:15.196 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672995 2024-04-25T09:05:15.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672995' 2024-04-25T09:05:15.196 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:05:15.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672995 -lt 42949672995 2024-04-25T09:05:15.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:15.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509472 2024-04-25T09:05:15.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:15.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:05:15.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509472 2024-04-25T09:05:15.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:15.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509472 2024-04-25T09:05:15.674 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509472 2024-04-25T09:05:15.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-04-25T09:05:15.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:05:16.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509473 -lt 64424509472 2024-04-25T09:05:16.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:16.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345944 2024-04-25T09:05:16.097 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:16.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:05:16.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345944 2024-04-25T09:05:16.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:16.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345944 2024-04-25T09:05:16.101 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345944 2024-04-25T09:05:16.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345944' 2024-04-25T09:05:16.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:05:16.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345944 -lt 85899345944 2024-04-25T09:05:16.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:16.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182416 2024-04-25T09:05:16.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:16.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:05:16.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182416 2024-04-25T09:05:16.519 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:16.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182416 2024-04-25T09:05:16.520 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182416 2024-04-25T09:05:16.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182416' 2024-04-25T09:05:16.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:05:17.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182417 -lt 107374182416 2024-04-25T09:05:17.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:17.012 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018886 2024-04-25T09:05:17.012 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:17.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:05:17.014 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018886 2024-04-25T09:05:17.014 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:17.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018886 2024-04-25T09:05:17.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018886' 2024-04-25T09:05:17.015 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018886 2024-04-25T09:05:17.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:05:17.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018887 -lt 128849018886 2024-04-25T09:05:17.440 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:05:17.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:17.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:18.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:05:18.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:05:18.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:05:18.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:05:18.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:05:18.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:05:18.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:05:18.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:05:18.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:05:18.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:05:18.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:18.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:05:18.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:05:18.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:05:18.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:05:18.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: seq 1 200 2024-04-25T09:05:18.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:18.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj1 /dev/null 2024-04-25T09:05:19.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:19.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj2 /dev/null 2024-04-25T09:05:19.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:19.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj3 /dev/null 2024-04-25T09:05:19.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:19.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj4 /dev/null 2024-04-25T09:05:19.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:19.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj5 /dev/null 2024-04-25T09:05:19.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:19.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj6 /dev/null 2024-04-25T09:05:19.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:19.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj7 /dev/null 2024-04-25T09:05:19.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:19.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj8 /dev/null 2024-04-25T09:05:19.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:19.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj9 /dev/null 2024-04-25T09:05:20.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:20.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj10 /dev/null 2024-04-25T09:05:20.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:20.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj11 /dev/null 2024-04-25T09:05:20.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:20.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj12 /dev/null 2024-04-25T09:05:20.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:20.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj13 /dev/null 2024-04-25T09:05:20.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:20.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj14 /dev/null 2024-04-25T09:05:20.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:20.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj15 /dev/null 2024-04-25T09:05:20.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:20.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj16 /dev/null 2024-04-25T09:05:20.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:20.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj17 /dev/null 2024-04-25T09:05:21.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:21.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj18 /dev/null 2024-04-25T09:05:21.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:21.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj19 /dev/null 2024-04-25T09:05:21.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:21.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj20 /dev/null 2024-04-25T09:05:21.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:21.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj21 /dev/null 2024-04-25T09:05:21.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:21.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj22 /dev/null 2024-04-25T09:05:21.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:21.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj23 /dev/null 2024-04-25T09:05:21.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:21.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj24 /dev/null 2024-04-25T09:05:21.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:21.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj25 /dev/null 2024-04-25T09:05:22.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:22.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj26 /dev/null 2024-04-25T09:05:22.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:22.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj27 /dev/null 2024-04-25T09:05:22.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:22.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj28 /dev/null 2024-04-25T09:05:22.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:22.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj29 /dev/null 2024-04-25T09:05:22.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:22.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj30 /dev/null 2024-04-25T09:05:22.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:22.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj31 /dev/null 2024-04-25T09:05:22.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:22.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj32 /dev/null 2024-04-25T09:05:22.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:22.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj33 /dev/null 2024-04-25T09:05:23.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:23.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj34 /dev/null 2024-04-25T09:05:23.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:23.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj35 /dev/null 2024-04-25T09:05:23.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:23.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj36 /dev/null 2024-04-25T09:05:23.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:23.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj37 /dev/null 2024-04-25T09:05:23.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:23.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj38 /dev/null 2024-04-25T09:05:23.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:23.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj39 /dev/null 2024-04-25T09:05:23.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:23.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj40 /dev/null 2024-04-25T09:05:23.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:23.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj41 /dev/null 2024-04-25T09:05:24.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:24.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj42 /dev/null 2024-04-25T09:05:24.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:24.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj43 /dev/null 2024-04-25T09:05:24.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:24.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj44 /dev/null 2024-04-25T09:05:24.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:24.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj45 /dev/null 2024-04-25T09:05:24.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:24.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj46 /dev/null 2024-04-25T09:05:24.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:24.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj47 /dev/null 2024-04-25T09:05:24.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:24.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj48 /dev/null 2024-04-25T09:05:25.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:25.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj49 /dev/null 2024-04-25T09:05:25.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:25.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj50 /dev/null 2024-04-25T09:05:25.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:25.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj51 /dev/null 2024-04-25T09:05:25.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:25.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj52 /dev/null 2024-04-25T09:05:25.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:25.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj53 /dev/null 2024-04-25T09:05:25.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:25.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj54 /dev/null 2024-04-25T09:05:25.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:25.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj55 /dev/null 2024-04-25T09:05:25.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:25.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj56 /dev/null 2024-04-25T09:05:25.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:25.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj57 /dev/null 2024-04-25T09:05:26.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:26.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj58 /dev/null 2024-04-25T09:05:26.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:26.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj59 /dev/null 2024-04-25T09:05:26.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:26.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj60 /dev/null 2024-04-25T09:05:26.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:26.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj61 /dev/null 2024-04-25T09:05:26.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:26.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj62 /dev/null 2024-04-25T09:05:26.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:26.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj63 /dev/null 2024-04-25T09:05:26.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:26.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj64 /dev/null 2024-04-25T09:05:26.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:26.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj65 /dev/null 2024-04-25T09:05:27.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:27.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj66 /dev/null 2024-04-25T09:05:27.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:27.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj67 /dev/null 2024-04-25T09:05:27.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:27.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj68 /dev/null 2024-04-25T09:05:27.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:27.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj69 /dev/null 2024-04-25T09:05:27.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:27.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj70 /dev/null 2024-04-25T09:05:27.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:27.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj71 /dev/null 2024-04-25T09:05:27.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:27.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj72 /dev/null 2024-04-25T09:05:27.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:27.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj73 /dev/null 2024-04-25T09:05:28.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:28.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj74 /dev/null 2024-04-25T09:05:28.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:28.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj75 /dev/null 2024-04-25T09:05:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj76 /dev/null 2024-04-25T09:05:28.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:28.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj77 /dev/null 2024-04-25T09:05:28.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:28.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj78 /dev/null 2024-04-25T09:05:28.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:28.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj79 /dev/null 2024-04-25T09:05:28.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:28.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj80 /dev/null 2024-04-25T09:05:28.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:28.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj81 /dev/null 2024-04-25T09:05:29.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:29.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj82 /dev/null 2024-04-25T09:05:29.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:29.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj83 /dev/null 2024-04-25T09:05:29.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:29.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj84 /dev/null 2024-04-25T09:05:29.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:29.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj85 /dev/null 2024-04-25T09:05:29.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:29.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj86 /dev/null 2024-04-25T09:05:29.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:29.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj87 /dev/null 2024-04-25T09:05:29.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:29.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj88 /dev/null 2024-04-25T09:05:29.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:29.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj89 /dev/null 2024-04-25T09:05:30.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:30.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj90 /dev/null 2024-04-25T09:05:30.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:30.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj91 /dev/null 2024-04-25T09:05:30.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:30.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj92 /dev/null 2024-04-25T09:05:30.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:30.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj93 /dev/null 2024-04-25T09:05:30.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:30.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj94 /dev/null 2024-04-25T09:05:30.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:30.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj95 /dev/null 2024-04-25T09:05:30.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:30.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj96 /dev/null 2024-04-25T09:05:30.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:30.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj97 /dev/null 2024-04-25T09:05:31.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:31.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj98 /dev/null 2024-04-25T09:05:31.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:31.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj99 /dev/null 2024-04-25T09:05:31.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:31.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj100 /dev/null 2024-04-25T09:05:31.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:31.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj101 /dev/null 2024-04-25T09:05:31.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:31.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj102 /dev/null 2024-04-25T09:05:31.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:31.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj103 /dev/null 2024-04-25T09:05:31.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:31.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj104 /dev/null 2024-04-25T09:05:31.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:31.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj105 /dev/null 2024-04-25T09:05:32.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:32.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj106 /dev/null 2024-04-25T09:05:32.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:32.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj107 /dev/null 2024-04-25T09:05:32.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:32.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj108 /dev/null 2024-04-25T09:05:32.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:32.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj109 /dev/null 2024-04-25T09:05:32.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:32.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj110 /dev/null 2024-04-25T09:05:32.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:32.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj111 /dev/null 2024-04-25T09:05:33.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:33.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj112 /dev/null 2024-04-25T09:05:33.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:33.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj113 /dev/null 2024-04-25T09:05:33.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:33.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj114 /dev/null 2024-04-25T09:05:33.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:33.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj115 /dev/null 2024-04-25T09:05:33.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:33.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj116 /dev/null 2024-04-25T09:05:33.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:33.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj117 /dev/null 2024-04-25T09:05:33.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:33.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj118 /dev/null 2024-04-25T09:05:34.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:34.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj119 /dev/null 2024-04-25T09:05:34.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:34.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj120 /dev/null 2024-04-25T09:05:34.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:34.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj121 /dev/null 2024-04-25T09:05:34.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:34.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj122 /dev/null 2024-04-25T09:05:34.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:34.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj123 /dev/null 2024-04-25T09:05:34.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:34.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj124 /dev/null 2024-04-25T09:05:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj125 /dev/null 2024-04-25T09:05:34.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:34.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj126 /dev/null 2024-04-25T09:05:35.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:35.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj127 /dev/null 2024-04-25T09:05:35.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:35.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj128 /dev/null 2024-04-25T09:05:35.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:35.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj129 /dev/null 2024-04-25T09:05:35.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:35.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj130 /dev/null 2024-04-25T09:05:35.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:35.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj131 /dev/null 2024-04-25T09:05:35.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:35.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj132 /dev/null 2024-04-25T09:05:35.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:35.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj133 /dev/null 2024-04-25T09:05:35.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:35.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj134 /dev/null 2024-04-25T09:05:36.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:36.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj135 /dev/null 2024-04-25T09:05:36.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:36.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj136 /dev/null 2024-04-25T09:05:36.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:36.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj137 /dev/null 2024-04-25T09:05:36.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:36.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj138 /dev/null 2024-04-25T09:05:36.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:36.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj139 /dev/null 2024-04-25T09:05:36.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:36.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj140 /dev/null 2024-04-25T09:05:36.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:36.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj141 /dev/null 2024-04-25T09:05:36.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:36.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj142 /dev/null 2024-04-25T09:05:37.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:37.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj143 /dev/null 2024-04-25T09:05:37.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:37.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj144 /dev/null 2024-04-25T09:05:37.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:37.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj145 /dev/null 2024-04-25T09:05:37.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:37.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj146 /dev/null 2024-04-25T09:05:37.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:37.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj147 /dev/null 2024-04-25T09:05:37.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:37.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj148 /dev/null 2024-04-25T09:05:37.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:37.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj149 /dev/null 2024-04-25T09:05:37.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:37.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj150 /dev/null 2024-04-25T09:05:38.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:38.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj151 /dev/null 2024-04-25T09:05:38.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:38.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj152 /dev/null 2024-04-25T09:05:38.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:38.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj153 /dev/null 2024-04-25T09:05:38.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:38.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj154 /dev/null 2024-04-25T09:05:38.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:38.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj155 /dev/null 2024-04-25T09:05:38.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:38.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj156 /dev/null 2024-04-25T09:05:38.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:38.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj157 /dev/null 2024-04-25T09:05:39.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:39.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj158 /dev/null 2024-04-25T09:05:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj159 /dev/null 2024-04-25T09:05:39.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:39.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj160 /dev/null 2024-04-25T09:05:39.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:39.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj161 /dev/null 2024-04-25T09:05:39.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:39.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj162 /dev/null 2024-04-25T09:05:39.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:39.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj163 /dev/null 2024-04-25T09:05:39.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:39.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj164 /dev/null 2024-04-25T09:05:39.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:39.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj165 /dev/null 2024-04-25T09:05:39.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:39.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj166 /dev/null 2024-04-25T09:05:40.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:40.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj167 /dev/null 2024-04-25T09:05:40.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:40.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj168 /dev/null 2024-04-25T09:05:40.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:40.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj169 /dev/null 2024-04-25T09:05:40.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:40.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj170 /dev/null 2024-04-25T09:05:40.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:40.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj171 /dev/null 2024-04-25T09:05:40.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:40.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj172 /dev/null 2024-04-25T09:05:40.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:40.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj173 /dev/null 2024-04-25T09:05:40.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:40.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj174 /dev/null 2024-04-25T09:05:41.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:41.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj175 /dev/null 2024-04-25T09:05:41.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:41.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj176 /dev/null 2024-04-25T09:05:41.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:41.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj177 /dev/null 2024-04-25T09:05:41.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:41.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj178 /dev/null 2024-04-25T09:05:41.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:41.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj179 /dev/null 2024-04-25T09:05:41.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:41.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj180 /dev/null 2024-04-25T09:05:41.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:41.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj181 /dev/null 2024-04-25T09:05:41.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:41.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj182 /dev/null 2024-04-25T09:05:42.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:42.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj183 /dev/null 2024-04-25T09:05:42.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:42.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj184 /dev/null 2024-04-25T09:05:42.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:42.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj185 /dev/null 2024-04-25T09:05:42.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:42.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj186 /dev/null 2024-04-25T09:05:42.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:42.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj187 /dev/null 2024-04-25T09:05:42.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:42.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj188 /dev/null 2024-04-25T09:05:42.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:42.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj189 /dev/null 2024-04-25T09:05:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj190 /dev/null 2024-04-25T09:05:43.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:43.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj191 /dev/null 2024-04-25T09:05:43.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:43.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj192 /dev/null 2024-04-25T09:05:43.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:43.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj193 /dev/null 2024-04-25T09:05:43.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:43.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj194 /dev/null 2024-04-25T09:05:43.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:43.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj195 /dev/null 2024-04-25T09:05:43.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:43.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj196 /dev/null 2024-04-25T09:05:43.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:43.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj197 /dev/null 2024-04-25T09:05:43.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:43.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj198 /dev/null 2024-04-25T09:05:44.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:44.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj199 /dev/null 2024-04-25T09:05:44.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:202: TEST_recovery_sizeup: for i in $(seq 1 $objects) 2024-04-25T09:05:44.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:204: TEST_recovery_sizeup: rados -p test put obj200 /dev/null 2024-04-25T09:05:44.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: get_primary test obj1 2024-04-25T09:05:44.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:05:44.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:05:44.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:05:44.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:05:44.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:207: TEST_recovery_sizeup: local primary=1 2024-04-25T09:05:44.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: get_pg test obj1 2024-04-25T09:05:44.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T09:05:44.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T09:05:44.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T09:05:44.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T09:05:45.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:208: TEST_recovery_sizeup: local PG=1.0 2024-04-25T09:05:45.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: get_not_primary test obj1 2024-04-25T09:05:45.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=test 2024-04-25T09:05:45.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=obj1 2024-04-25T09:05:45.273 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary test obj1 2024-04-25T09:05:45.273 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:05:45.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:05:45.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:05:45.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:05:45.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-04-25T09:05:45.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map test obj1 2024-04-25T09:05:45.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-04-25T09:05:46.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:210: TEST_recovery_sizeup: local otherosd=0 2024-04-25T09:05:46.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:212: TEST_recovery_sizeup: ceph osd set norecover 2024-04-25T09:05:47.013 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is set 2024-04-25T09:05:47.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:213: TEST_recovery_sizeup: ceph osd out osd.1 osd.0 2024-04-25T09:05:47.589 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already out. osd.0 is already out. 2024-04-25T09:05:47.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:214: TEST_recovery_sizeup: ceph osd pool set test size 4 2024-04-25T09:05:48.958 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 4 2024-04-25T09:05:48.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:215: TEST_recovery_sizeup: ceph osd unset norecover 2024-04-25T09:05:50.542 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is unset 2024-04-25T09:05:50.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: get_primary test obj1 2024-04-25T09:05:50.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:05:50.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:05:50.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:05:50.565 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:05:50.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:217: TEST_recovery_sizeup: primary=2 2024-04-25T09:05:50.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:219: TEST_recovery_sizeup: ceph tell osd.2 debug kick_recovery_wq 0 2024-04-25T09:05:51.148 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T09:05:51.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:220: TEST_recovery_sizeup: sleep 2 2024-04-25T09:05:53.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:222: TEST_recovery_sizeup: wait_for_clean 2024-04-25T09:05:53.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:05:53.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:05:53.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:05:53.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:05:53.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:05:53.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:05:53.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:05:53.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:05:53.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:05:53.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:05:53.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:05:53.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:05:53.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:05:53.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:05:53.292 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:05:53.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:05:53.752 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:05:53.752 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:05:53.752 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T09:05:53.752 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T09:05:53.752 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T09:05:53.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:05:53.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:53.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:05:53.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836537 2024-04-25T09:05:53.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836537 2024-04-25T09:05:53.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537' 2024-04-25T09:05:53.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:53.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:05:54.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673005 2024-04-25T09:05:54.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673005 2024-04-25T09:05:54.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673005' 2024-04-25T09:05:54.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:54.246 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:05:54.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509482 2024-04-25T09:05:54.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509482 2024-04-25T09:05:54.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673005 2-64424509482' 2024-04-25T09:05:54.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:54.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:05:54.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345954 2024-04-25T09:05:54.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345954 2024-04-25T09:05:54.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673005 2-64424509482 3-85899345954' 2024-04-25T09:05:54.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:54.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:05:54.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182427 2024-04-25T09:05:54.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182427 2024-04-25T09:05:54.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673005 2-64424509482 3-85899345954 4-107374182427' 2024-04-25T09:05:54.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:05:54.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:05:54.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018897 2024-04-25T09:05:54.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018897 2024-04-25T09:05:54.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836537 1-42949673005 2-64424509482 3-85899345954 4-107374182427 5-128849018897' 2024-04-25T09:05:54.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:54.910 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836537 2024-04-25T09:05:54.910 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:54.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:05:54.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836537 2024-04-25T09:05:54.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:54.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836537 2024-04-25T09:05:54.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2024-04-25T09:05:54.913 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836537 2024-04-25T09:05:54.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:05:55.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836537 -lt 21474836537 2024-04-25T09:05:55.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:55.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673005 2024-04-25T09:05:55.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:55.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:05:55.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673005 2024-04-25T09:05:55.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:55.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673005 2024-04-25T09:05:55.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-04-25T09:05:55.359 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673005 2024-04-25T09:05:55.360 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:05:55.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673003 -lt 42949673005 2024-04-25T09:05:55.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:05:56.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:05:56.863 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:05:57.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673005 -lt 42949673005 2024-04-25T09:05:57.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:57.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509482 2024-04-25T09:05:57.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:57.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:05:57.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509482 2024-04-25T09:05:57.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:57.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509482 2024-04-25T09:05:57.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509482' 2024-04-25T09:05:57.299 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509482 2024-04-25T09:05:57.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:05:57.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509483 -lt 64424509482 2024-04-25T09:05:57.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:57.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345954 2024-04-25T09:05:57.838 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:57.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:05:57.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345954 2024-04-25T09:05:57.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:57.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345954 2024-04-25T09:05:57.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345954' 2024-04-25T09:05:57.842 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345954 2024-04-25T09:05:57.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:05:58.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345954 -lt 85899345954 2024-04-25T09:05:58.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:58.421 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182427 2024-04-25T09:05:58.421 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:58.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:05:58.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:58.423 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182427 2024-04-25T09:05:58.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182427 2024-04-25T09:05:58.425 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182427 2024-04-25T09:05:58.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182427' 2024-04-25T09:05:58.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:05:58.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182427 -lt 107374182427 2024-04-25T09:05:58.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:05:58.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:05:58.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018897 2024-04-25T09:05:58.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:05:58.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018897 2024-04-25T09:05:58.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:05:58.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018897 2024-04-25T09:05:58.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018897' 2024-04-25T09:05:58.949 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018897 2024-04-25T09:05:58.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:05:59.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018897 -lt 128849018897 2024-04-25T09:05:59.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:05:59.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:05:59.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:00.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:06:00.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:00.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:00.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:00.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:00.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:00.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:00.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:00.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:00.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:00.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:00.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:06:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:06:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:06:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:01.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:01.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:01.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:01.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:01.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:01.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:01.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:01.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:01.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:01.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:01.722 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:01.722 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:01.722 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:02.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:02.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:02.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:02.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:02.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:02.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:02.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:02.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:02.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:03.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T09:06:03.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T09:06:03.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:03.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:03.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:03.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:03.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:03.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:03.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:03.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:03.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:03.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:03.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:03.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:03.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:03.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:03.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:04.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:04.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:04.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:04.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:04.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:04.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:04.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:04.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:04.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:04.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T09:06:04.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T09:06:04.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:04.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:04.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:05.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:05.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:05.051 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:05.051 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:05.051 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:05.051 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:05.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:05.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:05.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:05.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:05.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:05.537 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:06.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:06.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:06.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:06.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:06.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:06.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:06.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:06.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:06.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:06.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=5 2024-04-25T09:06:06.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 5 '!=' null 2024-04-25T09:06:06.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:06.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:06.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:06.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:06.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:06.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:06.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:06.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:06.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:06.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:06.904 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:07.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:07.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:07.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:07.386 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:08.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:08.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:08.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:08.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:08.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:08.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:08.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:08.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:08.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:08.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T09:06:08.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T09:06:08.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:08.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:08.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:08.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:08.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:08.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:08.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:08.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:08.717 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:08.718 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:08.718 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:09.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:09.213 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:09.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:09.214 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:09.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:09.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:09.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:09.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:09.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:09.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:09.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:09.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:09.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:10.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T09:06:10.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T09:06:10.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:10.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:10.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:10.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:10.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:10.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:10.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:10.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:10.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:10.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:10.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:10.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:10.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:10.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:10.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:11.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:11.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:11.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:11.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:11.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:11.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:11.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:11.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:11.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:12.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T09:06:12.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T09:06:12.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:12.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:12.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:12.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:12.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:12.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:12.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:12.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:12.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:12.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:12.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:12.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:12.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:12.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:12.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:13.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:13.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:13.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:13.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:13.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:13.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:13.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:13.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:13.250 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:13.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T09:06:13.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T09:06:13.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:13.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:13.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:13.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:13.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:13.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:13.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:13.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:13.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:13.910 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:13.910 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:14.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:14.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:14.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:14.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:15.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:15.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:15.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:15.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:15.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:15.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:15.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:15.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:15.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:15.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T09:06:15.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T09:06:15.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:15.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:15.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:15.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:15.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:15.864 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:15.864 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:15.864 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:15.864 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:15.865 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:15.865 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:16.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:16.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:16.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:16.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:16.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:16.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:16.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:16.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:16.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:16.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:16.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:16.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:16.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T09:06:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T09:06:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:17.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:17.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:17.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:17.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:17.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:17.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:17.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:17.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:18.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:18.205 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:18.205 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:18.205 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:18.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:18.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:18.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:18.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:18.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:18.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:18.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:18.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:18.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:19.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T09:06:19.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T09:06:19.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:19.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:19.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:19.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:19.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:19.601 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:19.601 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:19.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:19.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:19.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:19.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:20.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:20.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:20.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:20.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:20.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:20.722 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:20.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:21.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T09:06:21.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T09:06:21.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:21.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:21.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:21.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:21.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:21.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:21.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:21.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:21.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:21.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:21.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:21.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:21.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:21.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:21.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:22.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:22.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:22.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:22.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:22.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:22.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:22.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:22.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:22.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:22.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T09:06:22.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T09:06:22.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:22.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:22.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:23.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:23.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:23.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:23.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:23.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:23.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:23.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:23.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:23.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:23.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:23.478 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:23.478 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:24.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:24.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:24.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:24.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:24.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:24.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:24.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:24.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:24.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T09:06:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T09:06:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:24.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:24.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:24.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:24.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:24.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:24.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:24.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:24.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:25.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:25.378 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:25.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:25.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:26.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:26.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:26.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:26.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:26.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:26.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:26.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:26.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:26.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:26.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T09:06:26.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T09:06:26.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:26.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:26.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:26.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:26.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:26.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:26.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:26.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:26.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:26.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:26.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:27.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:27.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:27.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:27.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:28.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:28.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:28.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:28.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:28.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:28.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:28.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:28.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:28.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:28.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=4 2024-04-25T09:06:28.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 4 '!=' null 2024-04-25T09:06:28.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:28.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:28.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:28.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:28.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:28.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:28.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:28.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:28.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:28.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:28.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:29.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:29.203 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:29.203 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:29.203 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:29.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:29.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:30.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T09:06:30.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T09:06:30.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:30.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:30.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:30.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:30.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:30.645 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:30.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:30.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:30.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:30.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:30.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:31.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:31.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:31.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:31.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:31.803 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:31.803 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:32.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T09:06:32.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T09:06:32.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:32.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:32.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:32.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:32.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:32.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:32.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:32.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:32.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:32.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:32.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:32.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:32.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:32.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:32.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:33.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:33.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:33.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:33.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:33.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:33.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:33.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:33.463 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:33.463 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:34.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=6 2024-04-25T09:06:34.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 6 '!=' null 2024-04-25T09:06:34.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:34.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:34.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:34.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:34.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:34.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:34.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:34.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:34.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:34.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:34.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:34.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:06:34.604 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:34.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:34.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:35.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:06:35.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:06:35.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:06:35.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:06:35.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:06:35.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:06:35.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:06:35.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:06:35.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:06:35.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3 2024-04-25T09:06:35.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3 '!=' null 2024-04-25T09:06:35.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:06:35.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:06:35.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:06:35.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:06:35.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:06:35.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:06:35.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:06:35.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:06:35.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:06:35.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:06:35.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:06:36.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:06:36.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:06:36.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:06:36.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:06:36.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:06:36.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:06:36.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:06:36.794 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: expr 200 '*' 2 2024-04-25T09:06:36.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:224: TEST_recovery_sizeup: local degraded=400 2024-04-25T09:06:36.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: expr 200 '*' 2 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:225: TEST_recovery_sizeup: local misplaced=400 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:226: TEST_recovery_sizeup: local log=td/osd-recovery-stats/osd.2.log 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:227: TEST_recovery_sizeup: check td/osd-recovery-stats 1.0 2 replicated 400 0 400 0 200 0 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=2 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=400 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=0 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=400 2024-04-25T09:06:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T09:06:36.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start=200 2024-04-25T09:06:36.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end=0 2024-04-25T09:06:36.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.2.log 2024-04-25T09:06:36.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T09:06:36.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T09:06:36.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T09:06:36.800 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T09:06:36.800 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.2.log 2024-04-25T09:06:36.800 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T09:06:36.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[2,4,3,5]/[2,3]' 2024-04-25T09:06:36.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T09:06:36.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:06:36.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-04-25T09:06:36.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T09:06:36.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T09:06:36.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T09:06:36.832 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:06:36.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=400 2024-04-25T09:06:36.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 400 400 2024-04-25T09:06:36.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T09:06:36.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:06:36.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T09:06:36.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:06:36.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:06:36.837 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.2.log 2024-04-25T09:06:36.837 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T09:06:36.837 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:06:36.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=0 2024-04-25T09:06:36.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 0 0 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T09:06:36.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T09:06:36.851 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:06:36.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=400 2024-04-25T09:06:36.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 400 400 2024-04-25T09:06:36.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=400 2024-04-25T09:06:36.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:06:36.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=400 2024-04-25T09:06:36.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:06:36.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:06:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.2.log 2024-04-25T09:06:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T09:06:36.854 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:06:36.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T09:06:36.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T09:06:36.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T09:06:36.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:06:36.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T09:06:36.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:06:36.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n 200 ']' 2024-04-25T09:06:36.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:102: check: which='shard 2' 2024-04-25T09:06:36.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:06:36.867 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-04-25T09:06:36.868 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -F ' [2,4,3,5]/[2,3] ' 2024-04-25T09:06:36.868 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: grep -v est 2024-04-25T09:06:36.868 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: head -1 2024-04-25T09:06:36.868 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:103: check: FIRST=200 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:104: check: below_margin 200 200 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=200 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=200 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats shard 2 ' td/osd-recovery-stats/osd.2.log 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: tail -1 2024-04-25T09:06:36.871 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:06:36.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:105: check: LAST=0 2024-04-25T09:06:36.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:106: check: above_margin 0 0 2024-04-25T09:06:36.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T09:06:36.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:06:36.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T09:06:36.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:06:36.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:229: TEST_recovery_sizeup: delete_pool test 2024-04-25T09:06:36.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T09:06:36.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T09:06:37.401 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T09:06:37.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:230: TEST_recovery_sizeup: kill_daemons td/osd-recovery-stats 2024-04-25T09:06:37.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:06:37.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:06:37.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:06:37.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:06:37.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:06:42.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:06:42.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T09:06:42.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T09:06:42.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:06:42.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T09:06:42.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:06:42.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:06:42.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:06:42.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:06:42.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:06:42.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:06:42.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:06:42.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:06:42.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:06:42.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:06:42.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:06:42.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:06:42.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:06:42.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:06:42.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:06:42.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:06:42.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:06:42.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:06:42.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T09:06:42.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:06:42.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:42.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:06:42.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T09:06:42.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:06:42.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:36: run: for func in $funcs 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:37: run: setup td/osd-recovery-stats 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-recovery-stats 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-recovery-stats 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:06:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:06:42.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:06:42.803 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:06:42.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:06:42.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:06:42.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:06:42.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:06:42.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:06:42.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:06:42.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:06:42.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:06:42.806 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:06:42.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:06:42.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:06:42.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T09:06:42.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:06:42.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:42.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:06:42.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T09:06:42.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:06:42.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:06:42.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-recovery-stats 2024-04-25T09:06:42.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:06:42.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:42.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:06:42.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.280166 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-recovery-stats 1' TERM HUP INT 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:38: run: TEST_recovery_undersized td/osd-recovery-stats 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:306: TEST_recovery_undersized: local dir=td/osd-recovery-stats 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:308: TEST_recovery_undersized: local osds=3 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:309: TEST_recovery_undersized: run_mon td/osd-recovery-stats a 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-recovery-stats 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-recovery-stats/a 2024-04-25T09:06:42.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-recovery-stats/a --run-dir=td/osd-recovery-stats 2024-04-25T09:06:43.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:06:43.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:06:43.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:06:43.296 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:06:43.296 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:43.296 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:06:43.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:06:43.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-recovery-stats/a '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --mon-cluster-log-file=td/osd-recovery-stats/log --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:06:43.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:06:43.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:06:43.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:06:43.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:06:43.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:06:43.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:06:43.514 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:06:43.514 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:06:43.514 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:06:43.515 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:06:43.515 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:43.515 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:06:43.515 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T09:06:43.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:06:43.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get fsid 2024-04-25T09:06:43.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:06:43.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:06:43.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:06:43.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:06:43.632 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:06:43.633 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:06:43.633 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:06:43.633 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:06:43.633 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:06:43.633 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:43.633 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:06:43.633 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.280166/ceph-mon.a.asok 2024-04-25T09:06:43.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:06:43.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.280166/ceph-mon.a.asok config get mon_host 2024-04-25T09:06:43.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:310: TEST_recovery_undersized: run_mgr td/osd-recovery-stats x 2024-04-25T09:06:43.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-recovery-stats 2024-04-25T09:06:43.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:06:43.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:06:43.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:06:43.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-recovery-stats/x 2024-04-25T09:06:43.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:06:44.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:06:44.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:06:44.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:06:44.679 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:06:44.679 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:44.680 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:06:44.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:06:44.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:06:44.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-recovery-stats/x '--log-file=td/osd-recovery-stats/$name.log' '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --run-dir=td/osd-recovery-stats '--pid-file=td/osd-recovery-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:06:44.715 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: expr 3 - 1 2024-04-25T09:06:44.716 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: seq 0 2 2024-04-25T09:06:44.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T09:06:44.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 0 2024-04-25T09:06:44.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:06:44.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:06:44.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:06:44.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:06:44.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/0 2024-04-25T09:06:44.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:06:44.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/0' 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/0/journal' 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:06:44.721 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:06:44.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:06:44.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/0 2024-04-25T09:06:44.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:06:44.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d5602a43-70f9-4613-bc58-4a0dff49e095 2024-04-25T09:06:44.723 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 d5602a43-70f9-4613-bc58-4a0dff49e095 2024-04-25T09:06:44.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d5602a43-70f9-4613-bc58-4a0dff49e095' 2024-04-25T09:06:44.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:06:44.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAkHSpmwwrNLBAAV6+B3PFNEbS1siaj/taAqQ== 2024-04-25T09:06:44.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAkHSpmwwrNLBAAV6+B3PFNEbS1siaj/taAqQ=="}' 2024-04-25T09:06:44.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d5602a43-70f9-4613-bc58-4a0dff49e095 -i td/osd-recovery-stats/0/new.json 2024-04-25T09:06:45.109 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:06:45.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/0/new.json 2024-04-25T09:06:45.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAkHSpmwwrNLBAAV6+B3PFNEbS1siaj/taAqQ== --osd-uuid d5602a43-70f9-4613-bc58-4a0dff49e095 2024-04-25T09:06:45.161 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:45.173+0000 7efe1291d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:45.186 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:45.197+0000 7efe1291d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:45.188 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:45.197+0000 7efe1291d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:45.188 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:45.201+0000 7efe1291d7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T09:06:45.188 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:45.201+0000 7efe1291d7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T09:06:45.188 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:45.201+0000 7efe1291d7c0 -1 bluestore(td/osd-recovery-stats/0/block) _read_bdev_label failed to open td/osd-recovery-stats/0/block: (2) No such file or directory 2024-04-25T09:06:45.225 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:45.237+0000 7efe1291d7c0 -1 bluestore(td/osd-recovery-stats/0) _read_fsid unparsable uuid 2024-04-25T09:06:51.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/0/keyring 2024-04-25T09:06:51.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:06:51.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:06:52.085 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:06:52.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:06:52.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/0 --osd-journal=td/osd-recovery-stats/0/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:06:52.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:06:52.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:06:52.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:06:52.117 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:52.129+0000 7f67525f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:52.120 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:52.129+0000 7f67525f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:52.121 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:52.133+0000 7f67525f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:06:52.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:06:52.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:06:52.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:06:52.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:06:52.499 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:06:52.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:06:52.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:52.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:06:52.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:52.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:52.661 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:52.673+0000 7f67525f87c0 -1 Falling back to public interface 2024-04-25T09:06:52.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:53.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:53.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:53.939 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:06:53.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:06:53.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:53.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:54.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:55.344 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:06:55.357+0000 7f67525f87c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:06:55.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:55.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:55.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:06:55.349 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:06:55.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:55.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:55.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:56.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:56.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:56.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:06:56.854 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:06:56.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:56.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:57.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:58.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:58.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:58.357 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:06:58.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:06:58.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:58.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:06:58.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:06:59.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:06:59.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:06:59.787 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:06:59.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:06:59.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:06:59.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:00.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:01.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:01.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:01.315 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:07:01.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:07:01.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:01.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:01.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:02.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:02.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:02.745 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:07:02.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:07:02.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:02.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:03.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:04.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:04.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:04.316 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:07:04.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:07:04.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:04.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:04.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:05.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:05.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:05.789 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:07:05.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:07:05.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:05.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:06.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:07.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:07.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:07.304 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:07:07.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:07:07.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:07.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:07.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:08.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:08.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:08.742 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:07:08.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:07:08.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:08.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:09.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:10.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:10.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:10.193 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:07:10.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:07:10.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:10.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:11.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:12.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:12.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:12.037 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:07:12.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:07:12.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:12.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:13.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:14.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:14.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:14.197 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:07:14.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:07:14.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:14.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:14.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:15.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:15.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:15.946 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:07:15.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:07:15.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:15.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:16.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:17.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:17.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:17.828 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:07:17.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:07:17.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:17.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:18.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:19.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:19.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:19.378 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:07:19.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:07:19.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:19.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:20.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:21.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:21.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:21.005 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:07:21.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:07:21.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:21.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:07:21.424 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2893155463,v1:127.0.0.1:6803/2893155463] [v2:127.0.0.1:6804/2893155463,v1:127.0.0.1:6805/2893155463] exists,up d5602a43-70f9-4613-bc58-4a0dff49e095 2024-04-25T09:07:21.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:07:21.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:07:21.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:07:21.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 1 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/1 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/1' 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/1/journal' 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:07:21.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:07:21.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:07:21.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:07:21.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:07:21.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:07:21.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:07:21.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:07:21.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/1 2024-04-25T09:07:21.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:07:21.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e2b946e6-d8d7-43de-a964-d587c1ffde81 2024-04-25T09:07:21.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 e2b946e6-d8d7-43de-a964-d587c1ffde81' 2024-04-25T09:07:21.431 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 e2b946e6-d8d7-43de-a964-d587c1ffde81 2024-04-25T09:07:21.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:07:21.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBJHSpmzspLGxAAnD5TdVTKI0w9AsLWJhbHWg== 2024-04-25T09:07:21.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBJHSpmzspLGxAAnD5TdVTKI0w9AsLWJhbHWg=="}' 2024-04-25T09:07:21.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e2b946e6-d8d7-43de-a964-d587c1ffde81 -i td/osd-recovery-stats/1/new.json 2024-04-25T09:07:22.086 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:07:22.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/1/new.json 2024-04-25T09:07:22.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBJHSpmzspLGxAAnD5TdVTKI0w9AsLWJhbHWg== --osd-uuid e2b946e6-d8d7-43de-a964-d587c1ffde81 2024-04-25T09:07:22.139 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:22.149+0000 7fa59a83d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:22.171 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:22.181+0000 7fa59a83d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:22.172 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:22.185+0000 7fa59a83d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:22.173 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:22.185+0000 7fa59a83d7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T09:07:22.173 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:22.185+0000 7fa59a83d7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T09:07:22.173 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:22.185+0000 7fa59a83d7c0 -1 bluestore(td/osd-recovery-stats/1/block) _read_bdev_label failed to open td/osd-recovery-stats/1/block: (2) No such file or directory 2024-04-25T09:07:22.210 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:22.221+0000 7fa59a83d7c0 -1 bluestore(td/osd-recovery-stats/1) _read_fsid unparsable uuid 2024-04-25T09:07:28.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/1/keyring 2024-04-25T09:07:28.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:07:28.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:07:28.925 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:07:28.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:07:29.510 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:07:29.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:07:29.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:07:29.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:07:29.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:07:29.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/1 --osd-journal=td/osd-recovery-stats/1/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:07:29.543 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:29.552+0000 7fe942ba07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:29.559 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:29.568+0000 7fe942ba07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:29.560 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:29.572+0000 7fe942ba07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:07:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:07:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:07:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:07:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:07:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:07:29.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:29.927 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:07:29.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:07:29.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:29.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:30.110 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:30.120+0000 7fe942ba07c0 -1 Falling back to public interface 2024-04-25T09:07:30.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:31.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:31.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:31.350 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:07:31.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:07:31.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:31.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:31.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:32.718 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:07:32.728+0000 7fe942ba07c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:07:32.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:32.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:32.755 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:07:32.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:07:32.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:32.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:33.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:34.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:34.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:34.484 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:07:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:07:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:34.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:34.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:35.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:35.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:35.942 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:07:35.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:07:35.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:35.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:36.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:37.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:37.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:37.446 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:07:37.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:07:37.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:37.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:38.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:39.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:39.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:39.035 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:07:39.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:07:39.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:39.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:39.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:40.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:40.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:40.687 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:07:40.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:07:40.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:40.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:41.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:42.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:42.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:42.125 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:07:42.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:07:42.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:42.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:42.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:43.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:43.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:43.635 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:07:43.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:07:43.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:43.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:44.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:45.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:45.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:45.119 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:07:45.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:07:45.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:45.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:45.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:46.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:46.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:46.703 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:07:46.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:07:46.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:46.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:47.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:48.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:48.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:48.257 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:07:48.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:07:48.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:48.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:48.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:49.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:49.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:49.982 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:07:49.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:07:49.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:49.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:51.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:52.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:52.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:52.142 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:07:52.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:07:52.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:52.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:52.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:53.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:53.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:53.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:07:53.675 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:07:53.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:53.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:54.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:55.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:55.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:55.345 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:07:55.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:07:55.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:55.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:56.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:57.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:57.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:57.643 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:07:57.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:07:57.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:57.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:07:58.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:07:59.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:07:59.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:07:59.567 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:07:59.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:07:59.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:07:59.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:08:00.024 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/436822147,v1:127.0.0.1:6811/436822147] [v2:127.0.0.1:6812/436822147,v1:127.0.0.1:6813/436822147] exists,up e2b946e6-d8d7-43de-a964-d587c1ffde81 2024-04-25T09:08:00.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:08:00.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:08:00.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:311: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:313: TEST_recovery_undersized: run_osd td/osd-recovery-stats 2 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-recovery-stats 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-recovery-stats/2 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:08:00.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-recovery-stats/2' 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-recovery-stats/2/journal' 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-recovery-stats' 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:08:00.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-recovery-stats/$name.log' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-recovery-stats/$name.pid' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:08:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:08:00.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:08:00.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-recovery-stats/2 2024-04-25T09:08:00.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:08:00.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=765fd7b7-ec72-47ad-bfc2-12197198cb1e 2024-04-25T09:08:00.033 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 765fd7b7-ec72-47ad-bfc2-12197198cb1e 2024-04-25T09:08:00.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 765fd7b7-ec72-47ad-bfc2-12197198cb1e' 2024-04-25T09:08:00.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:08:00.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBwHSpmmqneAxAASb9UfUEyO+yeGAnoPv0xFA== 2024-04-25T09:08:00.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBwHSpmmqneAxAASb9UfUEyO+yeGAnoPv0xFA=="}' 2024-04-25T09:08:00.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 765fd7b7-ec72-47ad-bfc2-12197198cb1e -i td/osd-recovery-stats/2/new.json 2024-04-25T09:08:00.595 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:08:00.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-recovery-stats/2/new.json 2024-04-25T09:08:00.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBwHSpmmqneAxAASb9UfUEyO+yeGAnoPv0xFA== --osd-uuid 765fd7b7-ec72-47ad-bfc2-12197198cb1e 2024-04-25T09:08:00.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:00.648+0000 7fec2cbb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:00.657 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:00.668+0000 7fec2cbb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:00.659 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:00.668+0000 7fec2cbb37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:00.659 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:00.668+0000 7fec2cbb37c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T09:08:00.659 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:00.668+0000 7fec2cbb37c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T09:08:00.659 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:00.668+0000 7fec2cbb37c0 -1 bluestore(td/osd-recovery-stats/2/block) _read_bdev_label failed to open td/osd-recovery-stats/2/block: (2) No such file or directory 2024-04-25T09:08:00.773 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:00.784+0000 7fec2cbb37c0 -1 bluestore(td/osd-recovery-stats/2) _read_fsid unparsable uuid 2024-04-25T09:08:07.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-recovery-stats/2/keyring 2024-04-25T09:08:07.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:08:07.465 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:08:07.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:08:07.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-recovery-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:08:08.018 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:08:08.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:08:08.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=1ffc809e-a4f5-40db-bb28-2ed3fda78081 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-recovery-stats/2 --osd-journal=td/osd-recovery-stats/2/journal --chdir= --run-dir=td/osd-recovery-stats '--admin-socket=/tmp/ceph-asok.280166/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-recovery-stats/$name.log' '--pid-file=td/osd-recovery-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:08:08.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:08:08.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:08:08.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:08:08.045 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:08.056+0000 7f2a118897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:08.057 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:08.068+0000 7f2a118897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:08.058 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:08.068+0000 7f2a118897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:08:08.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:08:08.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:08:08.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:08:08.460 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:08:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:08:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:08:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:08:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:08.617 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:08.627+0000 7f2a118897c0 -1 Falling back to public interface 2024-04-25T09:08:08.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:09.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:09.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:09.880 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:08:09.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:08:09.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:09.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:10.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:11.264 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:08:11.275+0000 7f2a118897c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:08:11.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:11.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:11.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:08:11.291 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:08:11.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:11.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:12.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:13.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:13.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:13.004 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:08:13.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:08:13.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:13.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:13.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:14.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:14.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:14.440 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:08:14.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:08:14.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:14.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:14.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:15.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:15.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:15.853 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:08:15.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:08:15.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:15.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:16.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:17.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:17.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:17.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:08:17.565 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:08:17.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:17.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:18.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:19.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:19.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:19.071 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:08:19.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:08:19.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:19.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:19.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:20.482 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:08:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:08:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:20.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:21.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:21.904 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:08:21.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:21.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:08:21.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:21.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:22.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:23.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:23.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:23.477 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:08:23.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:08:23.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:23.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:24.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:25.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:25.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:25.028 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:08:25.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:08:25.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:25.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:25.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:26.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:26.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:26.995 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:08:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:08:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:26.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:27.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:28.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:28.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:28.670 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:08:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:08:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:28.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:29.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:30.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:08:30.571 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:08:30.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:30.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:31.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:32.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:32.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:32.237 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:08:32.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:08:32.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:32.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:33.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:34.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:34.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:34.423 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:08:34.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:08:34.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:34.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:35.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:08:36.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:08:36.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:08:36.224 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:08:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:08:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:08:36.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:08:36.659 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2405481861,v1:127.0.0.1:6819/2405481861] [v2:127.0.0.1:6820/2405481861,v1:127.0.0.1:6821/2405481861] exists,up 765fd7b7-ec72-47ad-bfc2-12197198cb1e 2024-04-25T09:08:36.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:08:36.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:08:36.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:08:36.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:316: TEST_recovery_undersized: create_pool test 1 1 2024-04-25T09:08:36.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T09:08:37.561 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T09:08:37.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:08:38.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:317: TEST_recovery_undersized: ceph osd pool set test size 1 --yes-i-really-mean-it 2024-04-25T09:08:39.669 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 1 2024-04-25T09:08:39.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:319: TEST_recovery_undersized: wait_for_clean 2024-04-25T09:08:39.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:08:39.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:08:39.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:08:39.693 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:08:39.694 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:08:39.694 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:08:39.694 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:08:39.694 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:08:39.694 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:08:39.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:08:39.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:08:39.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:08:39.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:08:39.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:08:39.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:08:40.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:08:40.402 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:08:40.402 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:08:40.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:08:40.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:08:40.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:08:40.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836500 2024-04-25T09:08:40.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836500 2024-04-25T09:08:40.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500' 2024-04-25T09:08:40.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:08:40.574 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:08:40.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672977 2024-04-25T09:08:40.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672977 2024-04-25T09:08:40.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977' 2024-04-25T09:08:40.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:08:40.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:08:40.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509448 2024-04-25T09:08:40.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509448 2024-04-25T09:08:40.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836500 1-42949672977 2-64424509448' 2024-04-25T09:08:40.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:08:40.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836500 2024-04-25T09:08:40.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:08:40.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:08:40.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836500 2024-04-25T09:08:40.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:08:40.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836500 2024-04-25T09:08:40.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-04-25T09:08:40.857 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836500 2024-04-25T09:08:40.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:08:41.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836498 -lt 21474836500 2024-04-25T09:08:41.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:08:42.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:08:42.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:08:42.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836501 -lt 21474836500 2024-04-25T09:08:42.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:08:42.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672977 2024-04-25T09:08:42.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:08:42.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:08:42.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672977 2024-04-25T09:08:42.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:08:42.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672977 2024-04-25T09:08:42.681 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672977 2024-04-25T09:08:42.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672977' 2024-04-25T09:08:42.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:08:43.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672977 -lt 42949672977 2024-04-25T09:08:43.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:08:43.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509448 2024-04-25T09:08:43.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:08:43.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:08:43.134 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:08:43.134 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509448 2024-04-25T09:08:43.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509448 2024-04-25T09:08:43.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509448' 2024-04-25T09:08:43.135 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509448 2024-04-25T09:08:43.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:08:43.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509449 -lt 64424509448 2024-04-25T09:08:43.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:08:43.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:08:43.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:08:44.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:08:44.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:08:44.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:08:44.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:08:44.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:08:44.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:08:44.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:08:44.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:08:44.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:08:44.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:08:44.478 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:08:44.478 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:08:45.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:08:45.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:08:45.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:08:45.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: seq 1 200 2024-04-25T09:08:45.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj1 /dev/null 2024-04-25T09:08:45.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj2 /dev/null 2024-04-25T09:08:45.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj3 /dev/null 2024-04-25T09:08:45.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj4 /dev/null 2024-04-25T09:08:45.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj5 /dev/null 2024-04-25T09:08:45.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj6 /dev/null 2024-04-25T09:08:45.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj7 /dev/null 2024-04-25T09:08:45.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj8 /dev/null 2024-04-25T09:08:45.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj9 /dev/null 2024-04-25T09:08:45.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj10 /dev/null 2024-04-25T09:08:45.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:45.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj11 /dev/null 2024-04-25T09:08:46.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj12 /dev/null 2024-04-25T09:08:46.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj13 /dev/null 2024-04-25T09:08:46.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj14 /dev/null 2024-04-25T09:08:46.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj15 /dev/null 2024-04-25T09:08:46.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj16 /dev/null 2024-04-25T09:08:46.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj17 /dev/null 2024-04-25T09:08:46.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj18 /dev/null 2024-04-25T09:08:46.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj19 /dev/null 2024-04-25T09:08:46.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj20 /dev/null 2024-04-25T09:08:46.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj21 /dev/null 2024-04-25T09:08:46.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:46.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj22 /dev/null 2024-04-25T09:08:47.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj23 /dev/null 2024-04-25T09:08:47.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj24 /dev/null 2024-04-25T09:08:47.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj25 /dev/null 2024-04-25T09:08:47.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj26 /dev/null 2024-04-25T09:08:47.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj27 /dev/null 2024-04-25T09:08:47.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj28 /dev/null 2024-04-25T09:08:47.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj29 /dev/null 2024-04-25T09:08:47.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj30 /dev/null 2024-04-25T09:08:47.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj31 /dev/null 2024-04-25T09:08:47.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj32 /dev/null 2024-04-25T09:08:47.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj33 /dev/null 2024-04-25T09:08:47.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:47.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj34 /dev/null 2024-04-25T09:08:48.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj35 /dev/null 2024-04-25T09:08:48.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj36 /dev/null 2024-04-25T09:08:48.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj37 /dev/null 2024-04-25T09:08:48.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj38 /dev/null 2024-04-25T09:08:48.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj39 /dev/null 2024-04-25T09:08:48.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj40 /dev/null 2024-04-25T09:08:48.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj41 /dev/null 2024-04-25T09:08:48.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj42 /dev/null 2024-04-25T09:08:48.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj43 /dev/null 2024-04-25T09:08:48.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj44 /dev/null 2024-04-25T09:08:48.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:48.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj45 /dev/null 2024-04-25T09:08:49.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj46 /dev/null 2024-04-25T09:08:49.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj47 /dev/null 2024-04-25T09:08:49.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj48 /dev/null 2024-04-25T09:08:49.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj49 /dev/null 2024-04-25T09:08:49.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj50 /dev/null 2024-04-25T09:08:49.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj51 /dev/null 2024-04-25T09:08:49.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj52 /dev/null 2024-04-25T09:08:49.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj53 /dev/null 2024-04-25T09:08:49.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj54 /dev/null 2024-04-25T09:08:49.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:49.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj55 /dev/null 2024-04-25T09:08:50.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj56 /dev/null 2024-04-25T09:08:50.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj57 /dev/null 2024-04-25T09:08:50.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj58 /dev/null 2024-04-25T09:08:50.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj59 /dev/null 2024-04-25T09:08:50.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj60 /dev/null 2024-04-25T09:08:50.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj61 /dev/null 2024-04-25T09:08:50.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj62 /dev/null 2024-04-25T09:08:50.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj63 /dev/null 2024-04-25T09:08:50.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj64 /dev/null 2024-04-25T09:08:50.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj65 /dev/null 2024-04-25T09:08:50.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj66 /dev/null 2024-04-25T09:08:50.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:50.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj67 /dev/null 2024-04-25T09:08:51.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj68 /dev/null 2024-04-25T09:08:51.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj69 /dev/null 2024-04-25T09:08:51.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj70 /dev/null 2024-04-25T09:08:51.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj71 /dev/null 2024-04-25T09:08:51.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj72 /dev/null 2024-04-25T09:08:51.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj73 /dev/null 2024-04-25T09:08:51.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj74 /dev/null 2024-04-25T09:08:51.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj75 /dev/null 2024-04-25T09:08:51.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj76 /dev/null 2024-04-25T09:08:51.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj77 /dev/null 2024-04-25T09:08:51.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:51.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj78 /dev/null 2024-04-25T09:08:52.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj79 /dev/null 2024-04-25T09:08:52.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj80 /dev/null 2024-04-25T09:08:52.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj81 /dev/null 2024-04-25T09:08:52.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj82 /dev/null 2024-04-25T09:08:52.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj83 /dev/null 2024-04-25T09:08:52.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj84 /dev/null 2024-04-25T09:08:52.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj85 /dev/null 2024-04-25T09:08:52.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj86 /dev/null 2024-04-25T09:08:52.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj87 /dev/null 2024-04-25T09:08:52.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj88 /dev/null 2024-04-25T09:08:52.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj89 /dev/null 2024-04-25T09:08:52.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:52.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj90 /dev/null 2024-04-25T09:08:53.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj91 /dev/null 2024-04-25T09:08:53.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj92 /dev/null 2024-04-25T09:08:53.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj93 /dev/null 2024-04-25T09:08:53.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj94 /dev/null 2024-04-25T09:08:53.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj95 /dev/null 2024-04-25T09:08:53.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj96 /dev/null 2024-04-25T09:08:53.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj97 /dev/null 2024-04-25T09:08:53.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj98 /dev/null 2024-04-25T09:08:53.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj99 /dev/null 2024-04-25T09:08:53.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj100 /dev/null 2024-04-25T09:08:53.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:53.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj101 /dev/null 2024-04-25T09:08:54.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj102 /dev/null 2024-04-25T09:08:54.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj103 /dev/null 2024-04-25T09:08:54.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj104 /dev/null 2024-04-25T09:08:54.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj105 /dev/null 2024-04-25T09:08:54.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj106 /dev/null 2024-04-25T09:08:54.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj107 /dev/null 2024-04-25T09:08:54.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj108 /dev/null 2024-04-25T09:08:54.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj109 /dev/null 2024-04-25T09:08:54.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj110 /dev/null 2024-04-25T09:08:54.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj111 /dev/null 2024-04-25T09:08:54.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj112 /dev/null 2024-04-25T09:08:54.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj113 /dev/null 2024-04-25T09:08:54.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:54.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj114 /dev/null 2024-04-25T09:08:55.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj115 /dev/null 2024-04-25T09:08:55.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj116 /dev/null 2024-04-25T09:08:55.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj117 /dev/null 2024-04-25T09:08:55.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj118 /dev/null 2024-04-25T09:08:55.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj119 /dev/null 2024-04-25T09:08:55.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj120 /dev/null 2024-04-25T09:08:55.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj121 /dev/null 2024-04-25T09:08:55.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj122 /dev/null 2024-04-25T09:08:55.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj123 /dev/null 2024-04-25T09:08:55.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj124 /dev/null 2024-04-25T09:08:55.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj125 /dev/null 2024-04-25T09:08:55.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:55.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj126 /dev/null 2024-04-25T09:08:56.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj127 /dev/null 2024-04-25T09:08:56.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj128 /dev/null 2024-04-25T09:08:56.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj129 /dev/null 2024-04-25T09:08:56.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj130 /dev/null 2024-04-25T09:08:56.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj131 /dev/null 2024-04-25T09:08:56.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj132 /dev/null 2024-04-25T09:08:56.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj133 /dev/null 2024-04-25T09:08:56.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj134 /dev/null 2024-04-25T09:08:56.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj135 /dev/null 2024-04-25T09:08:56.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj136 /dev/null 2024-04-25T09:08:56.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj137 /dev/null 2024-04-25T09:08:56.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:56.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj138 /dev/null 2024-04-25T09:08:57.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj139 /dev/null 2024-04-25T09:08:57.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj140 /dev/null 2024-04-25T09:08:57.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj141 /dev/null 2024-04-25T09:08:57.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj142 /dev/null 2024-04-25T09:08:57.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj143 /dev/null 2024-04-25T09:08:57.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj144 /dev/null 2024-04-25T09:08:57.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj145 /dev/null 2024-04-25T09:08:57.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj146 /dev/null 2024-04-25T09:08:57.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj147 /dev/null 2024-04-25T09:08:57.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj148 /dev/null 2024-04-25T09:08:57.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj149 /dev/null 2024-04-25T09:08:57.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:57.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj150 /dev/null 2024-04-25T09:08:58.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj151 /dev/null 2024-04-25T09:08:58.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj152 /dev/null 2024-04-25T09:08:58.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj153 /dev/null 2024-04-25T09:08:58.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj154 /dev/null 2024-04-25T09:08:58.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj155 /dev/null 2024-04-25T09:08:58.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj156 /dev/null 2024-04-25T09:08:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj157 /dev/null 2024-04-25T09:08:58.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj158 /dev/null 2024-04-25T09:08:58.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj159 /dev/null 2024-04-25T09:08:58.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj160 /dev/null 2024-04-25T09:08:58.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj161 /dev/null 2024-04-25T09:08:58.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj162 /dev/null 2024-04-25T09:08:58.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:58.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj163 /dev/null 2024-04-25T09:08:59.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj164 /dev/null 2024-04-25T09:08:59.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj165 /dev/null 2024-04-25T09:08:59.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj166 /dev/null 2024-04-25T09:08:59.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj167 /dev/null 2024-04-25T09:08:59.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj168 /dev/null 2024-04-25T09:08:59.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj169 /dev/null 2024-04-25T09:08:59.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj170 /dev/null 2024-04-25T09:08:59.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj171 /dev/null 2024-04-25T09:08:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj172 /dev/null 2024-04-25T09:08:59.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj173 /dev/null 2024-04-25T09:08:59.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:08:59.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj174 /dev/null 2024-04-25T09:09:00.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj175 /dev/null 2024-04-25T09:09:00.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj176 /dev/null 2024-04-25T09:09:00.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj177 /dev/null 2024-04-25T09:09:00.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj178 /dev/null 2024-04-25T09:09:00.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj179 /dev/null 2024-04-25T09:09:00.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj180 /dev/null 2024-04-25T09:09:00.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj181 /dev/null 2024-04-25T09:09:00.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj182 /dev/null 2024-04-25T09:09:00.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj183 /dev/null 2024-04-25T09:09:00.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj184 /dev/null 2024-04-25T09:09:00.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj185 /dev/null 2024-04-25T09:09:00.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj186 /dev/null 2024-04-25T09:09:00.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:00.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj187 /dev/null 2024-04-25T09:09:01.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj188 /dev/null 2024-04-25T09:09:01.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj189 /dev/null 2024-04-25T09:09:01.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj190 /dev/null 2024-04-25T09:09:01.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj191 /dev/null 2024-04-25T09:09:01.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj192 /dev/null 2024-04-25T09:09:01.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj193 /dev/null 2024-04-25T09:09:01.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj194 /dev/null 2024-04-25T09:09:01.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj195 /dev/null 2024-04-25T09:09:01.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj196 /dev/null 2024-04-25T09:09:01.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj197 /dev/null 2024-04-25T09:09:01.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj198 /dev/null 2024-04-25T09:09:01.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:01.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj199 /dev/null 2024-04-25T09:09:02.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:321: TEST_recovery_undersized: for i in $(seq 1 $objects) 2024-04-25T09:09:02.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:323: TEST_recovery_undersized: rados -p test put obj200 /dev/null 2024-04-25T09:09:02.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: get_primary test obj1 2024-04-25T09:09:02.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:09:02.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:09:02.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:09:02.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:09:02.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:326: TEST_recovery_undersized: local primary=1 2024-04-25T09:09:02.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: get_pg test obj1 2024-04-25T09:09:02.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=test 2024-04-25T09:09:02.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj1 2024-04-25T09:09:02.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map test obj1 2024-04-25T09:09:02.546 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T09:09:02.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:327: TEST_recovery_undersized: local PG=1.0 2024-04-25T09:09:02.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:329: TEST_recovery_undersized: ceph osd set norecover 2024-04-25T09:09:03.838 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is set 2024-04-25T09:09:03.860 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: expr 3 - 1 2024-04-25T09:09:03.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: seq 0 2 2024-04-25T09:09:03.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:331: TEST_recovery_undersized: for i in $(seq 0 $(expr $osds - 1)) 2024-04-25T09:09:03.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:333: TEST_recovery_undersized: '[' 0 = 1 ']' 2024-04-25T09:09:03.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:337: TEST_recovery_undersized: ceph osd out osd.0 2024-04-25T09:09:04.524 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T09:09:04.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:338: TEST_recovery_undersized: break 2024-04-25T09:09:04.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:340: TEST_recovery_undersized: ceph osd pool set test size 4 2024-04-25T09:09:05.483 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 4 2024-04-25T09:09:05.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:341: TEST_recovery_undersized: ceph osd unset norecover 2024-04-25T09:09:06.500 INFO:tasks.workunit.client.0.smithi134.stderr:norecover is unset 2024-04-25T09:09:06.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: get_primary test obj1 2024-04-25T09:09:06.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:09:06.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:09:06.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:09:06.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:09:07.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:342: TEST_recovery_undersized: ceph tell osd.1 debug kick_recovery_wq 0 2024-04-25T09:09:07.284 INFO:tasks.workunit.client.0.smithi134.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2024-04-25T09:09:07.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:344: TEST_recovery_undersized: sleep 10 2024-04-25T09:09:17.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:345: TEST_recovery_undersized: flush_pg_stats 2024-04-25T09:09:17.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:09:17.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:09:17.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:09:17.758 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:09:17.758 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:09:17.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:09:17.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:17.758 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:09:17.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836510 2024-04-25T09:09:17.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836510 2024-04-25T09:09:17.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510' 2024-04-25T09:09:17.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:17.965 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:09:18.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672986 2024-04-25T09:09:18.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672986 2024-04-25T09:09:18.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672986' 2024-04-25T09:09:18.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:09:18.167 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:09:18.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509458 2024-04-25T09:09:18.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509458 2024-04-25T09:09:18.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836510 1-42949672986 2-64424509458' 2024-04-25T09:09:18.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:18.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836510 2024-04-25T09:09:18.320 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:18.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:09:18.321 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836510 2024-04-25T09:09:18.321 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:18.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836510 2024-04-25T09:09:18.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-04-25T09:09:18.323 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836510 2024-04-25T09:09:18.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:18.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836508 -lt 21474836510 2024-04-25T09:09:18.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:09:19.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:09:19.734 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:09:20.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836510 2024-04-25T09:09:20.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:20.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672986 2024-04-25T09:09:20.148 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:20.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:09:20.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672986 2024-04-25T09:09:20.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:20.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672986 2024-04-25T09:09:20.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672986' 2024-04-25T09:09:20.152 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672986 2024-04-25T09:09:20.153 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:09:20.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672987 -lt 42949672986 2024-04-25T09:09:20.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:09:20.598 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509458 2024-04-25T09:09:20.598 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:09:20.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:09:20.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509458 2024-04-25T09:09:20.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:09:20.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509458 2024-04-25T09:09:20.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509458' 2024-04-25T09:09:20.601 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509458 2024-04-25T09:09:20.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:09:21.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509458 -lt 64424509458 2024-04-25T09:09:21.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: seq 1 300 2024-04-25T09:09:21.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:350: TEST_recovery_undersized: for i in $(seq 1 300) 2024-04-25T09:09:21.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: ceph pg dump pgs 2024-04-25T09:09:21.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep -qv recovering 2024-04-25T09:09:21.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:352: TEST_recovery_undersized: grep '^1.0' 2024-04-25T09:09:21.420 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T09:09:21.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:354: TEST_recovery_undersized: break 2024-04-25T09:09:21.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: get_primary test obj1 2024-04-25T09:09:21.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=test 2024-04-25T09:09:21.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=obj1 2024-04-25T09:09:21.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map test obj1 2024-04-25T09:09:21.437 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-04-25T09:09:21.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:365: TEST_recovery_undersized: primary=1 2024-04-25T09:09:21.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:366: TEST_recovery_undersized: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T09:09:21.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: expr 200 '*' 3 2024-04-25T09:09:21.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:368: TEST_recovery_undersized: local first_degraded=600 2024-04-25T09:09:21.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: expr 200 '*' 2 2024-04-25T09:09:21.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:369: TEST_recovery_undersized: local last_degraded=400 2024-04-25T09:09:21.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:370: TEST_recovery_undersized: check td/osd-recovery-stats 1.0 1 replicated 600 400 0 0 2024-04-25T09:09:21.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:64: check: local dir=td/osd-recovery-stats 2024-04-25T09:09:21.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:65: check: local PG=1.0 2024-04-25T09:09:21.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:66: check: local primary=1 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:67: check: local type=replicated 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:68: check: local degraded_start=600 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:69: check: local degraded_end=400 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:70: check: local misplaced_start=0 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:71: check: local misplaced_end=0 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:72: check: local primary_start= 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:73: check: local primary_end= 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:75: check: local log=td/osd-recovery-stats/osd.1.log 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:77: check: local 'addp= ' 2024-04-25T09:09:21.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:78: check: '[' replicated = erasure ']' 2024-04-25T09:09:21.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '"' '$log' '|' tail -1 '|' sed '"s/.*[)]' '\([[][^' 'p]*\).*$/\1/"' 2024-04-25T09:09:21.842 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:09:21.842 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: tail -1 2024-04-25T09:09:21.842 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: sed 's/.*[)] \([[][^ p]*\).*$/\1/' 2024-04-25T09:09:21.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:83: check: UPACT='[1,2]' 2024-04-25T09:09:21.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:86: check: local which=degraded 2024-04-25T09:09:21.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:09:21.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:09:21.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -F ' [1,2] ' 2024-04-25T09:09:21.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: grep -v est 2024-04-25T09:09:21.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: head -1 2024-04-25T09:09:21.851 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:09:21.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:87: check: FIRST=600 2024-04-25T09:09:21.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:88: check: below_margin 600 600 2024-04-25T09:09:21.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=600 2024-04-25T09:09:21.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:09:21.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=600 2024-04-25T09:09:21.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:09:21.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:09:21.857 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats degraded ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:09:21.857 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: tail -1 2024-04-25T09:09:21.857 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:09:21.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:89: check: LAST=400 2024-04-25T09:09:21.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:90: check: above_margin 400 400 2024-04-25T09:09:21.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=400 2024-04-25T09:09:21.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:09:21.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=400 2024-04-25T09:09:21.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:09:21.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:93: check: which=misplaced 2024-04-25T09:09:21.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' grep -F '"' '${UPACT}${addp}"' '|' grep -v est '|' head -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:09:21.866 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:09:21.866 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -F ' [1,2] ' 2024-04-25T09:09:21.866 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: grep -v est 2024-04-25T09:09:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: head -1 2024-04-25T09:09:21.867 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:09:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:94: check: FIRST=0 2024-04-25T09:09:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:95: check: below_margin 0 0 2024-04-25T09:09:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:44: below_margin: local -i check=0 2024-04-25T09:09:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:45: below_margin: shift 2024-04-25T09:09:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:46: below_margin: local -i target=0 2024-04-25T09:09:21.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:48: below_margin: return 0 2024-04-25T09:09:21.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: eval grep '"pg[[]${PG}.*recovering.*PeeringState::update_calc_stats' '$which' '"' '$log' '|' tail -1 '|' sed '"s/.*' '\([0-9]*\)$/\1/"' 2024-04-25T09:09:21.871 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: grep 'pg[[]1.0.*recovering.*PeeringState::update_calc_stats misplaced ' td/osd-recovery-stats/osd.1.log 2024-04-25T09:09:21.871 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: tail -1 2024-04-25T09:09:21.871 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: sed 's/.* \([0-9]*\)$/\1/' 2024-04-25T09:09:21.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:96: check: LAST=0 2024-04-25T09:09:21.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:97: check: above_margin 0 0 2024-04-25T09:09:21.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:52: above_margin: local -i check=0 2024-04-25T09:09:21.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:53: above_margin: shift 2024-04-25T09:09:21.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:54: above_margin: local -i target=0 2024-04-25T09:09:21.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:56: above_margin: return 0 2024-04-25T09:09:21.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:100: check: '[' -n '' ']' 2024-04-25T09:09:21.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:372: TEST_recovery_undersized: delete_pool test 2024-04-25T09:09:21.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T09:09:21.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T09:09:22.399 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T09:09:22.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:373: TEST_recovery_undersized: kill_daemons td/osd-recovery-stats 2024-04-25T09:09:22.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:09:22.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:09:22.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:09:22.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:09:22.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:09:27.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:09:27.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-recovery-stats.sh:39: run: teardown td/osd-recovery-stats 2024-04-25T09:09:27.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T09:09:27.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:09:27.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T09:09:27.735 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:09:27.735 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:09:27.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:09:27.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:09:27.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:09:27.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:09:27.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:09:27.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:09:27.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:09:27.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:09:27.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:09:27.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:09:27.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:09:27.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:09:27.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:09:27.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:09:27.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:09:27.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:09:27.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T09:09:27.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:09:27.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:27.780 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:09:27.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T09:09:27.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:09:27.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:09:27.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:09:27.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-recovery-stats 0 2024-04-25T09:09:27.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-recovery-stats 2024-04-25T09:09:27.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:09:27.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-recovery-stats KILL 2024-04-25T09:09:27.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:09:27.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:09:27.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:09:27.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:09:27.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:09:27.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:09:27.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:09:27.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:09:27.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:09:27.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:09:27.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:09:27.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:09:27.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:09:27.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:09:27.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:09:27.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:09:27.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:09:27.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:09:27.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-recovery-stats 2024-04-25T09:09:27.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:09:27.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:27.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.280166 2024-04-25T09:09:27.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.280166 2024-04-25T09:09:27.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:09:27.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:09:27.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:09:27.795 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:09:27.795 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:09:27.851 INFO:tasks.workunit:Running workunit osd/osd-rep-recov-eio.sh... 2024-04-25T09:09:27.852 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-rep-recov-eio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh 2024-04-25T09:09:27.908 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:09:27.914 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-rep-recov-eio.sh 2024-04-25T09:09:27.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:25: run: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:27.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:26: run: shift 2024-04-25T09:09:27.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: export CEPH_MON=127.0.0.1:7140 2024-04-25T09:09:27.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:28: run: CEPH_MON=127.0.0.1:7140 2024-04-25T09:09:27.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:29: run: export CEPH_ARGS 2024-04-25T09:09:27.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: uuidgen 2024-04-25T09:09:27.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:30: run: CEPH_ARGS+='--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none ' 2024-04-25T09:09:27.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:31: run: CEPH_ARGS+='--mon-host=127.0.0.1:7140 ' 2024-04-25T09:09:27.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: set 2024-04-25T09:09:27.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:09:27.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:34: run: local 'funcs=TEST_rados_get_with_eio 2024-04-25T09:09:27.919 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_rados_repair_warning 2024-04-25T09:09:27.919 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_rep_backfill_unfound 2024-04-25T09:09:27.919 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_rep_recovery_unfound' 2024-04-25T09:09:27.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:09:27.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:09:27.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:27.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:09:27.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:27.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:09:27.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:09:27.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:09:27.921 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:09:27.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:09:27.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:09:27.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:09:27.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:09:27.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:09:27.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:09:27.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:09:27.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:09:27.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:09:27.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:09:27.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:09:27.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:09:27.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:09:27.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:09:27.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:09:27.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:09:27.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:09:27.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:09:27.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:27.929 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:09:27.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.390086 2024-04-25T09:09:27.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:09:27.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:09:27.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:09:27.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:09:27.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:27.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:09:27.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.390086 2024-04-25T09:09:27.934 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:09:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:09:28.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:09:28.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:09:28.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:09:28.433 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:09:28.433 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:28.433 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:09:28.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:09:28.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:09:28.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:09:28.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:09:28.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:09:28.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:09:28.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:09:28.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:09:28.630 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:09:28.630 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:09:28.630 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:09:28.631 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:09:28.631 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:28.631 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:09:28.631 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-mon.a.asok 2024-04-25T09:09:28.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:09:28.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.390086/ceph-mon.a.asok config get fsid 2024-04-25T09:09:28.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:09:28.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:09:28.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:09:28.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:09:28.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:09:28.750 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:09:28.750 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:09:28.751 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:09:28.751 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:09:28.751 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:28.751 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:09:28.751 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-mon.a.asok 2024-04-25T09:09:28.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:09:28.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.390086/ceph-mon.a.asok config get mon_host 2024-04-25T09:09:28.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:09:28.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:28.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:09:28.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:09:28.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:09:28.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:09:28.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:09:29.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:09:29.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:09:29.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:09:29.792 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:09:29.792 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:29.792 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:09:29.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:09:29.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:09:29.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:09:29.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:09:30.164 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T09:09:30.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_get_with_eio td/osd-rep-recov-eio.sh 2024-04-25T09:09:30.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:168: TEST_rados_get_with_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:30.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:170: TEST_rados_get_with_eio: setup_osds 4 2024-04-25T09:09:30.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-04-25T09:09:30.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:09:30.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:09:30.182 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-04-25T09:09:30.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-04-25T09:09:30.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:09:30.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:09:30.188 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:09:30.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:09:30.190 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:09:30.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8acd0bf9-25ea-4404-92f6-ff02cb2fe45f 2024-04-25T09:09:30.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8acd0bf9-25ea-4404-92f6-ff02cb2fe45f' 2024-04-25T09:09:30.190 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 8acd0bf9-25ea-4404-92f6-ff02cb2fe45f 2024-04-25T09:09:30.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:09:30.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDKHSpmMgYCDRAAEU/2gd7DUzG5wG2p0BhySA== 2024-04-25T09:09:30.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDKHSpmMgYCDRAAEU/2gd7DUzG5wG2p0BhySA=="}' 2024-04-25T09:09:30.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8acd0bf9-25ea-4404-92f6-ff02cb2fe45f -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:09:30.540 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:09:30.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:09:30.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDKHSpmMgYCDRAAEU/2gd7DUzG5wG2p0BhySA== --osd-uuid 8acd0bf9-25ea-4404-92f6-ff02cb2fe45f 2024-04-25T09:09:30.588 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:30.602+0000 7f1173f987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:30.609 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:30.622+0000 7f1173f987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:30.610 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:30.622+0000 7f1173f987c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:30.610 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:30.622+0000 7f1173f987c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:09:30.610 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:30.622+0000 7f1173f987c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:09:30.610 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:30.622+0000 7f1173f987c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:09:30.648 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:30.662+0000 7f1173f987c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:09:36.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:09:36.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:09:36.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:09:36.894 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:09:36.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:09:37.435 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:09:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:09:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:09:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:09:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:09:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:09:37.465 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:37.477+0000 7fbcef96a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:37.494 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:37.477+0000 7fbcef96a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:37.494 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:37.481+0000 7fbcef96a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:09:37.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:09:37.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:09:37.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:09:37.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:09:37.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:09:37.845 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:09:37.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:37.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:09:37.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:37.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:38.009 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:38.021+0000 7fbcef96a7c0 -1 Falling back to public interface 2024-04-25T09:09:38.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:39.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:39.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:39.254 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:09:39.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:09:39.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:39.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:39.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:40.710 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:09:40.721+0000 7fbcef96a7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:09:40.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:40.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:40.721 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:09:40.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:09:40.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:40.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:41.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:42.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:42.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:42.228 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:09:42.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:09:42.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:42.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:42.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:43.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:43.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:43.678 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:09:43.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:09:43.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:43.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:44.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:45.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:45.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:45.118 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:09:45.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:09:45.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:45.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:45.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:46.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:46.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:46.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:09:46.688 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:09:46.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:46.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:47.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:48.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:48.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:48.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:09:48.150 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:09:48.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:48.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:48.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:49.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:49.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:49.667 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:09:49.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:09:49.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:49.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:50.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:51.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:51.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:51.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:09:51.132 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:09:51.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:51.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:51.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:52.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:52.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:52.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:09:52.680 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:09:52.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:52.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:53.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:54.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:54.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:54.116 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:09:54.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:09:54.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:54.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:54.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:55.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:55.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:55.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:09:55.577 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:09:55.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:55.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:56.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:57.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:57.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:57.255 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:09:57.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:09:57.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:57.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:58.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:09:59.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:09:59.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:09:59.203 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:09:59.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:09:59.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:09:59.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:09:59.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:00.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:00.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:00.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:10:00.707 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:10:00.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:00.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:01.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:02.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:02.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:02.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:10:02.584 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:10:02.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:02.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:03.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:04.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:04.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:04.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:10:04.781 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:10:04.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:04.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:05.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:06.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:06.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:06.672 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:10:06.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:10:06.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:06.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:10:07.152 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/538825488,v1:127.0.0.1:6803/538825488] [v2:127.0.0.1:6804/538825488,v1:127.0.0.1:6805/538825488] exists,up 8acd0bf9-25ea-4404-92f6-ff02cb2fe45f 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:10:07.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:10:07.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:10:07.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:10:07.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:10:07.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=31dbc289-cf12-46bf-ba7d-2cd1678d7921 2024-04-25T09:10:07.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 31dbc289-cf12-46bf-ba7d-2cd1678d7921' 2024-04-25T09:10:07.159 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 31dbc289-cf12-46bf-ba7d-2cd1678d7921 2024-04-25T09:10:07.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:10:07.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvHSpmPaxXCxAAkz12t6k88JZYRW7wiAI64g== 2024-04-25T09:10:07.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvHSpmPaxXCxAAkz12t6k88JZYRW7wiAI64g=="}' 2024-04-25T09:10:07.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 31dbc289-cf12-46bf-ba7d-2cd1678d7921 -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:10:07.649 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:10:07.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:10:07.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvHSpmPaxXCxAAkz12t6k88JZYRW7wiAI64g== --osd-uuid 31dbc289-cf12-46bf-ba7d-2cd1678d7921 2024-04-25T09:10:07.697 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:07.709+0000 7f9b194b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:07.734 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:07.745+0000 7f9b194b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:07.736 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:07.749+0000 7f9b194b77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:07.737 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:07.749+0000 7f9b194b77c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:10:07.737 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:07.749+0000 7f9b194b77c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:10:07.737 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:07.749+0000 7f9b194b77c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:10:07.773 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:07.785+0000 7f9b194b77c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:10:14.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:10:14.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:10:14.467 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:10:14.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:10:14.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:10:15.010 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:10:15.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:10:15.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:10:15.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:10:15.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:10:15.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:10:15.039 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:15.049+0000 7f6364b587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:15.041 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:15.053+0000 7f6364b587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:15.042 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:15.053+0000 7f6364b587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:15.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:10:15.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:10:15.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:10:15.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:10:15.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:10:15.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:15.448 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:10:15.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:10:15.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:15.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:15.605 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:15.617+0000 7f6364b587c0 -1 Falling back to public interface 2024-04-25T09:10:15.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:16.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:16.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:16.875 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:10:16.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:10:16.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:16.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:17.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:18.200 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:18.213+0000 7f6364b587c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:10:18.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:18.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:18.300 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:10:18.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:10:18.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:18.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:18.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:19.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:19.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:19.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:10:19.923 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:10:19.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:19.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:20.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:21.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:21.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:21.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:10:21.397 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:10:21.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:21.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:21.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:22.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:22.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:22.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:10:22.877 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:10:22.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:22.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:23.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:24.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:24.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:24.434 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:10:24.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:10:24.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:24.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:24.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:25.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:25.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:25.849 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:10:25.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:10:25.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:25.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:26.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:27.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:27.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:27.411 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:10:27.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:10:27.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:27.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:27.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:28.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:28.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:28.874 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:10:28.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:10:28.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:28.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:29.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:30.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:30.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:30.367 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:10:30.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:10:30.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:30.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:30.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:31.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:31.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:31.874 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:10:31.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:10:31.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:31.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:32.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:33.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:33.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:33.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:10:33.626 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:10:33.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:33.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:34.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:35.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:35.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:35.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:10:35.342 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:35.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:36.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:37.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:37.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:37.131 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:10:37.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:10:37.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:37.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:38.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:39.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:39.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:39.075 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:10:39.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:10:39.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:39.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:39.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:40.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:40.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:40.983 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:10:40.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:10:40.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:40.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:41.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:42.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:42.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:42.555 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:10:42.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:10:42.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:42.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:43.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:10:44.360 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:10:44.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:44.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:10:44.846 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 13 up_thru 13 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/539324914,v1:127.0.0.1:6811/539324914] [v2:127.0.0.1:6812/539324914,v1:127.0.0.1:6813/539324914] exists,up 31dbc289-cf12-46bf-ba7d-2cd1678d7921 2024-04-25T09:10:44.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:10:44.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:10:44.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:10:44.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:10:44.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:10:44.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:10:44.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:10:44.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:10:44.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:10:44.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:10:44.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:10:44.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:10:44.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6975a61d-624b-4dd1-9841-7a4517d296d9 2024-04-25T09:10:44.853 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 6975a61d-624b-4dd1-9841-7a4517d296d9 2024-04-25T09:10:44.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6975a61d-624b-4dd1-9841-7a4517d296d9' 2024-04-25T09:10:44.854 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:10:44.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAUHipmkluTNBAA+QFZc4A0T9q23toEOTk8Lw== 2024-04-25T09:10:44.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAUHipmkluTNBAA+QFZc4A0T9q23toEOTk8Lw=="}' 2024-04-25T09:10:44.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6975a61d-624b-4dd1-9841-7a4517d296d9 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:10:45.491 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:10:45.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:10:45.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAUHipmkluTNBAA+QFZc4A0T9q23toEOTk8Lw== --osd-uuid 6975a61d-624b-4dd1-9841-7a4517d296d9 2024-04-25T09:10:45.541 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:45.552+0000 7fcdba5157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:45.544 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:45.556+0000 7fcdba5157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:45.545 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:45.556+0000 7fcdba5157c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:45.545 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:45.556+0000 7fcdba5157c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:10:45.546 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:45.556+0000 7fcdba5157c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:10:45.546 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:45.556+0000 7fcdba5157c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:10:45.648 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:45.660+0000 7fcdba5157c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:10:52.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:10:52.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:10:52.126 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:10:52.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:10:52.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:10:52.707 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:10:52.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:10:52.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:10:52.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:10:52.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:10:52.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:10:52.736 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:52.748+0000 7fb44055c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:52.738 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:52.748+0000 7fb44055c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:52.740 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:52.752+0000 7fb44055c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:10:53.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:10:53.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:10:53.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:10:53.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:10:53.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:10:53.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:53.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:10:53.116 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:10:53.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:53.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:10:53.285 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:53.296+0000 7fb44055c7c0 -1 Falling back to public interface 2024-04-25T09:10:53.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:54.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:54.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:54.538 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:10:54.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:10:54.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:54.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:10:54.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:55.903 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:10:55.912+0000 7fb44055c7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:10:55.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:55.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:55.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:10:55.958 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:10:55.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:55.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:10:56.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:57.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:57.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:57.795 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:10:57.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:10:57.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:57.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:10:58.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:10:59.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:10:59.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:10:59.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:10:59.344 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:10:59.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:10:59.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:10:59.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:00.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:00.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:00.931 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:11:00.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:11:00.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:00.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:01.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:02.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:02.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:02.572 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:11:02.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:11:02.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:02.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:03.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:04.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:04.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:04.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:11:04.002 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:11:04.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:04.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:04.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:05.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:05.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:05.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:11:05.452 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:11:05.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:05.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:06.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:07.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:07.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:07.002 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:11:07.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:11:07.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:07.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:07.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:08.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:08.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:08.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:11:08.538 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:11:08.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:08.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:09.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:10.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:10.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:10.103 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:11:10.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:11:10.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:10.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:10.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:11.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:11.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:11.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:11:11.505 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:11:11.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:11.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:12.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:13.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:13.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:13.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:11:13.078 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:11:13.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:13.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:13.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:14.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:14.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:14.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:11:14.960 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:11:14.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:14.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:15.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:16.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:16.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:16.999 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:11:17.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:11:17.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:17.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:18.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:19.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:19.495 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:11:19.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:19.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:11:19.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:19.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:20.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:21.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:21.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:21.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:11:21.168 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:11:21.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:21.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:22.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:23.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:23.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:23.089 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:11:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:11:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:23.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:24.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:25.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:25.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:25.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:11:25.119 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T09:11:25.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:25.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:11:25.911 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 21 up_thru 21 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1054361274,v1:127.0.0.1:6819/1054361274] [v2:127.0.0.1:6820/1054361274,v1:127.0.0.1:6821/1054361274] exists,up 6975a61d-624b-4dd1-9841-7a4517d296d9 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:11:25.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:11:25.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:11:25.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-04-25T09:11:25.918 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:11:25.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bb0aeb49-2ea1-42ed-9ba3-76e332478323 2024-04-25T09:11:25.918 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 bb0aeb49-2ea1-42ed-9ba3-76e332478323 2024-04-25T09:11:25.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 bb0aeb49-2ea1-42ed-9ba3-76e332478323' 2024-04-25T09:11:25.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:11:25.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9Hipm4/u0OBAAhQf/N1fFsvkkaZ6GhW5QDA== 2024-04-25T09:11:25.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9Hipm4/u0OBAAhQf/N1fFsvkkaZ6GhW5QDA=="}' 2024-04-25T09:11:25.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bb0aeb49-2ea1-42ed-9ba3-76e332478323 -i td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:11:26.496 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:11:26.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:11:26.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9Hipm4/u0OBAAhQf/N1fFsvkkaZ6GhW5QDA== --osd-uuid bb0aeb49-2ea1-42ed-9ba3-76e332478323 2024-04-25T09:11:26.545 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:26.555+0000 7fdb352fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:26.547 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:26.559+0000 7fdb352fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:26.548 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:26.559+0000 7fdb352fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:26.549 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:26.559+0000 7fdb352fe7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:11:26.549 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:26.559+0000 7fdb352fe7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:11:26.549 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:26.559+0000 7fdb352fe7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:11:26.647 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:26.655+0000 7fdb352fe7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-04-25T09:11:33.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-04-25T09:11:33.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:11:33.842 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T09:11:33.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:11:33.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:11:34.402 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T09:11:34.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:11:34.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:11:34.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:11:34.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:11:34.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:11:34.431 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:34.439+0000 7f2ce0df17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:34.433 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:34.443+0000 7f2ce0df17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:34.434 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:34.443+0000 7f2ce0df17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:11:34.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:11:34.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:11:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:11:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:11:34.839 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:11:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:11:34.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:34.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:11:34.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:34.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:35.013 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:35.023+0000 7f2ce0df17c0 -1 Falling back to public interface 2024-04-25T09:11:35.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:36.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:36.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:36.278 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:11:36.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:11:36.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:36.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:36.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:37.642 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:11:37.651+0000 7f2ce0df17c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:11:37.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:37.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:37.718 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:11:37.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:11:37.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:37.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:38.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:39.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:39.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:39.445 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:11:39.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:11:39.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:39.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:40.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:41.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:41.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:41.267 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:11:41.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:11:41.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:41.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:41.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:42.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:42.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:42.860 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:11:42.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:11:42.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:42.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:43.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:44.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:44.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:44.460 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:11:44.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:11:44.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:44.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:44.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:45.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:45.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:45.971 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:11:45.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:11:45.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:45.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:46.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:47.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:47.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:47.622 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:11:47.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:11:47.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:47.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:48.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:49.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:49.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:49.103 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:11:49.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:11:49.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:49.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:49.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:50.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:50.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:50.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:11:50.610 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:11:50.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:50.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:51.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:52.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:52.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:52.020 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:11:52.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:11:52.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:52.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:52.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:53.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:53.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:53.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:11:53.429 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:11:53.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:53.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:55.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:55.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:55.253 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:11:55.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:11:55.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:55.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:56.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:57.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:57.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:57.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:11:57.398 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:11:57.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:57.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:11:58.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:11:59.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:11:59.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:11:59.209 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:11:59.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:11:59.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:11:59.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:12:00.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:01.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:01.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:01.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:12:01.131 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:12:01.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:01.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:12:01.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:02.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:02.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:02.837 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:12:02.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:12:02.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:02.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:12:04.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:12:05.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:12:05.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:12:05.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:12:05.201 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:12:05.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:12:05.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:12:05.739 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/3701627663,v1:127.0.0.1:6827/3701627663] [v2:127.0.0.1:6828/3701627663,v1:127.0.0.1:6829/3701627663] exists,up bb0aeb49-2ea1-42ed-9ba3-76e332478323 2024-04-25T09:12:05.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:12:05.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:12:05.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:12:05.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:12:05.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:12:05.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:12:05.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:12:05.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:12:05.741 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:12:05.741 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:12:05.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:12:05.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:12:05.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:12:05.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:12:05.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:12:05.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:12:05.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:12:05.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:12:05.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:12:06.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:12:06.530 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:06.530 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:12:06.530 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:12:06.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:12:06.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:06.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:12:06.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803810 2024-04-25T09:12:06.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803810 2024-04-25T09:12:06.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810' 2024-04-25T09:12:06.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:06.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:12:06.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574875 2024-04-25T09:12:06.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574875 2024-04-25T09:12:06.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810 1-55834574875' 2024-04-25T09:12:06.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:06.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:12:07.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313233 2024-04-25T09:12:07.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313233 2024-04-25T09:12:07.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810 1-55834574875 2-90194313233' 2024-04-25T09:12:07.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:07.212 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:12:07.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051594 2024-04-25T09:12:07.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051594 2024-04-25T09:12:07.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803810 1-55834574875 2-90194313233 3-124554051594' 2024-04-25T09:12:07.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:07.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803810 2024-04-25T09:12:07.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:07.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:12:07.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803810 2024-04-25T09:12:07.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:07.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803810 2024-04-25T09:12:07.370 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803810 2024-04-25T09:12:07.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803810' 2024-04-25T09:12:07.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:07.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803808 -lt 25769803810 2024-04-25T09:12:07.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:12:08.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:12:08.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:09.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803810 -lt 25769803810 2024-04-25T09:12:09.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:09.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574875 2024-04-25T09:12:09.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:09.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:12:09.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574875 2024-04-25T09:12:09.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:09.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574875 2024-04-25T09:12:09.372 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574875 2024-04-25T09:12:09.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574875' 2024-04-25T09:12:09.372 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:09.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574875 -lt 55834574875 2024-04-25T09:12:09.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:09.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313233 2024-04-25T09:12:09.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:09.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:12:09.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313233 2024-04-25T09:12:09.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:09.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313233 2024-04-25T09:12:09.815 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 90194313233 2024-04-25T09:12:09.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313233' 2024-04-25T09:12:09.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:12:10.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313233 -lt 90194313233 2024-04-25T09:12:10.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:10.238 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-124554051594 2024-04-25T09:12:10.238 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:10.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:12:10.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-124554051594 2024-04-25T09:12:10.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:10.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051594 2024-04-25T09:12:10.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 124554051594' 2024-04-25T09:12:10.241 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 124554051594 2024-04-25T09:12:10.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:12:10.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051595 -lt 124554051594 2024-04-25T09:12:10.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:12:10.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:10.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:11.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:12:11.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:12:11.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:12:11.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:12:11.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:12:11.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:12:11.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:12:11.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:12:11.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=6 2024-04-25T09:12:11.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:12:11.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:11.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:12.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 6 = 8 2024-04-25T09:12:12.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 6 '!=' -1 2024-04-25T09:12:12.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:12:12.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=6 2024-04-25T09:12:12.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:12:12.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:12:12.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:12:12.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:12:12.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:12:12.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:12:12.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:12:12.325 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:12:12.326 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:12:12.326 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:12:12.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:12:12.770 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:12:12.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:12.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:13.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:12:13.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:12:13.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:12:13.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:172: TEST_rados_get_with_eio: local poolname=pool-rep 2024-04-25T09:12:13.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:173: TEST_rados_get_with_eio: create_pool pool-rep 1 1 2024-04-25T09:12:13.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-04-25T09:12:13.821 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'pool-rep' already exists 2024-04-25T09:12:13.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:12:14.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:174: TEST_rados_get_with_eio: wait_for_clean 2024-04-25T09:12:14.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:12:14.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:12:14.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:12:14.840 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:12:14.841 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:12:14.841 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:12:14.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:12:14.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:12:14.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:12:14.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:12:14.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:12:14.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:12:14.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:12:14.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:12:14.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:12:15.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:12:15.496 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:15.496 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:12:15.496 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:12:15.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:12:15.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:15.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:12:15.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803813 2024-04-25T09:12:15.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803813 2024-04-25T09:12:15.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813' 2024-04-25T09:12:15.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:15.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:12:15.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574879 2024-04-25T09:12:15.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574879 2024-04-25T09:12:15.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574879' 2024-04-25T09:12:15.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:15.877 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:12:16.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313237 2024-04-25T09:12:16.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313237 2024-04-25T09:12:16.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574879 2-90194313237' 2024-04-25T09:12:16.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:16.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:12:16.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051598 2024-04-25T09:12:16.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051598 2024-04-25T09:12:16.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803813 1-55834574879 2-90194313237 3-124554051598' 2024-04-25T09:12:16.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:16.166 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803813 2024-04-25T09:12:16.167 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:16.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:12:16.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803813 2024-04-25T09:12:16.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:16.170 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803813 2024-04-25T09:12:16.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803813 2024-04-25T09:12:16.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803813' 2024-04-25T09:12:16.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:16.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803814 -lt 25769803813 2024-04-25T09:12:16.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:16.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574879 2024-04-25T09:12:16.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:16.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:12:16.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574879 2024-04-25T09:12:16.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:16.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574879 2024-04-25T09:12:16.601 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574879 2024-04-25T09:12:16.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574879' 2024-04-25T09:12:16.601 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:17.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574879 -lt 55834574879 2024-04-25T09:12:17.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:17.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313237 2024-04-25T09:12:17.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:17.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:12:17.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313237 2024-04-25T09:12:17.036 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:17.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313237 2024-04-25T09:12:17.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313237' 2024-04-25T09:12:17.037 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 90194313237 2024-04-25T09:12:17.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:12:17.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313237 -lt 90194313237 2024-04-25T09:12:17.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:17.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-124554051598 2024-04-25T09:12:17.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:17.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:12:17.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-124554051598 2024-04-25T09:12:17.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:17.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051598 2024-04-25T09:12:17.468 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 124554051598 2024-04-25T09:12:17.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 124554051598' 2024-04-25T09:12:17.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:12:17.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051598 -lt 124554051598 2024-04-25T09:12:17.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:12:17.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:17.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:18.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:12:18.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:12:18.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:12:18.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:12:18.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:12:18.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:12:18.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:12:18.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:12:18.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:12:18.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:12:18.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:18.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:19.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:12:19.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:12:19.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:12:19.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:175: TEST_rados_get_with_eio: rados_get_data eio td/osd-rep-recov-eio.sh 2024-04-25T09:12:19.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:111: rados_get_data: local inject=eio 2024-04-25T09:12:19.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:112: rados_get_data: shift 2024-04-25T09:12:19.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:113: rados_get_data: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:19.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:115: rados_get_data: local poolname=pool-rep 2024-04-25T09:12:19.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:116: rados_get_data: local objname=obj-eio-390086 2024-04-25T09:12:19.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: get_pg pool-rep obj-eio-390086 2024-04-25T09:12:19.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-04-25T09:12:19.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-eio-390086 2024-04-25T09:12:19.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:19.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T09:12:19.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:117: rados_get_data: local pgid=2.0 2024-04-25T09:12:19.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:119: rados_get_data: rados_put td/osd-rep-recov-eio.sh pool-rep obj-eio-390086 2024-04-25T09:12:19.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:19.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:12:19.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-eio-390086 2024-04-25T09:12:19.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:12:19.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:12:19.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:12:19.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:12:19.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:12:19.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:12:19.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:12:19.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:12:19.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-eio-390086 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:12:20.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:120: rados_get_data: inject_eio rep data pool-rep obj-eio-390086 td/osd-rep-recov-eio.sh 0 2024-04-25T09:12:20.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:12:20.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:12:20.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:12:20.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:12:20.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:12:20.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:12:20.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-390086 2024-04-25T09:12:20.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:12:20.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:20.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:12:20.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:12:20.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:12:20.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-390086 2024-04-25T09:12:20.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:12:20.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-390086 2024-04-25T09:12:20.060 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:20.060 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:12:20.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:12:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:12:20.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:12:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:12:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:12:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:12:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:12:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:12:20.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:12:20.485 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:12:20.486 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:20.486 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:20.486 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:20.486 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:12:20.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:12:20.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:12:20.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:12:20.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:12:20.603 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:12:20.603 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:12:20.603 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:12:20.603 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:20.603 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:20.603 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:20.604 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:12:20.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:12:20.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-eio-390086 2024-04-25T09:12:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:121: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-390086 2024-04-25T09:12:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:12:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-390086 2024-04-25T09:12:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:12:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:12:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:12:20.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-390086 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:20.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:20.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:20.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:123: rados_get_data: wait_for_clean 2024-04-25T09:12:20.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:12:20.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:12:20.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:12:20.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:12:20.825 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:12:20.825 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:12:20.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:12:20.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:12:20.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:12:20.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:12:20.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:12:20.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:12:20.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:12:20.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:12:20.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:12:21.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:12:21.369 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:21.369 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:12:21.369 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:12:21.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:12:21.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:21.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:12:21.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803817 2024-04-25T09:12:21.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803817 2024-04-25T09:12:21.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817' 2024-04-25T09:12:21.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:21.542 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:12:21.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574882 2024-04-25T09:12:21.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574882 2024-04-25T09:12:21.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-55834574882' 2024-04-25T09:12:21.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:21.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:12:21.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313240 2024-04-25T09:12:21.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313240 2024-04-25T09:12:21.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-55834574882 2-90194313240' 2024-04-25T09:12:21.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:21.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:12:22.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051601 2024-04-25T09:12:22.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051601 2024-04-25T09:12:22.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-55834574882 2-90194313240 3-124554051601' 2024-04-25T09:12:22.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:22.014 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803817 2024-04-25T09:12:22.014 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:22.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:12:22.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803817 2024-04-25T09:12:22.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:22.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803817 2024-04-25T09:12:22.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-04-25T09:12:22.017 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803817 2024-04-25T09:12:22.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:22.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803814 -lt 25769803817 2024-04-25T09:12:22.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:12:23.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:12:23.426 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:23.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803817 -lt 25769803817 2024-04-25T09:12:23.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:23.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574882 2024-04-25T09:12:23.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:23.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:12:23.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574882 2024-04-25T09:12:23.839 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:23.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574882 2024-04-25T09:12:23.841 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574882 2024-04-25T09:12:23.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574882' 2024-04-25T09:12:23.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:24.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574882 -lt 55834574882 2024-04-25T09:12:24.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:24.326 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313240 2024-04-25T09:12:24.326 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:24.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:12:24.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313240 2024-04-25T09:12:24.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:24.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313240 2024-04-25T09:12:24.330 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 90194313240 2024-04-25T09:12:24.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313240' 2024-04-25T09:12:24.330 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:12:24.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313240 -lt 90194313240 2024-04-25T09:12:24.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:24.736 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-124554051601 2024-04-25T09:12:24.736 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:24.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:12:24.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-124554051601 2024-04-25T09:12:24.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:24.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051601 2024-04-25T09:12:24.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 124554051601' 2024-04-25T09:12:24.740 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 124554051601 2024-04-25T09:12:24.740 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:12:25.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051601 -lt 124554051601 2024-04-25T09:12:25.146 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:12:25.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:25.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:25.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:12:25.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:12:25.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:12:25.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:12:25.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:12:25.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:12:25.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:12:25.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:12:26.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:12:26.078 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:12:26.079 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:26.079 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:26.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:12:26.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:12:26.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:12:26.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: ceph pg 2.0 query 2024-04-25T09:12:26.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:12:26.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:124: rados_get_data: COUNT=1 2024-04-25T09:12:26.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:125: rados_get_data: test 1 = 1 2024-04-25T09:12:26.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:126: rados_get_data: flush_pg_stats 2024-04-25T09:12:26.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:12:26.783 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:12:27.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:12:27.211 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:27.211 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:12:27.211 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:12:27.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:12:27.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:27.211 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:12:27.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803820 2024-04-25T09:12:27.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803820 2024-04-25T09:12:27.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820' 2024-04-25T09:12:27.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:27.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:12:27.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574885 2024-04-25T09:12:27.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574885 2024-04-25T09:12:27.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-55834574885' 2024-04-25T09:12:27.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:27.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:12:27.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313243 2024-04-25T09:12:27.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313243 2024-04-25T09:12:27.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-55834574885 2-90194313243' 2024-04-25T09:12:27.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:27.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:12:27.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051604 2024-04-25T09:12:27.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051604 2024-04-25T09:12:27.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-55834574885 2-90194313243 3-124554051604' 2024-04-25T09:12:27.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:27.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803820 2024-04-25T09:12:27.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:27.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:12:27.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803820 2024-04-25T09:12:27.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:27.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803820 2024-04-25T09:12:27.845 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803820 2024-04-25T09:12:27.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2024-04-25T09:12:27.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:28.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803818 -lt 25769803820 2024-04-25T09:12:28.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:12:29.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:12:29.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:29.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803820 -lt 25769803820 2024-04-25T09:12:29.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:29.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:29.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574885 2024-04-25T09:12:29.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:12:29.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:29.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574885 2024-04-25T09:12:29.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574885 2024-04-25T09:12:29.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574885' 2024-04-25T09:12:29.660 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574885 2024-04-25T09:12:29.660 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:30.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574885 -lt 55834574885 2024-04-25T09:12:30.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:30.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313243 2024-04-25T09:12:30.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:30.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:12:30.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313243 2024-04-25T09:12:30.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:30.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313243 2024-04-25T09:12:30.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313243' 2024-04-25T09:12:30.088 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 90194313243 2024-04-25T09:12:30.088 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:12:30.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313243 -lt 90194313243 2024-04-25T09:12:30.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:30.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-124554051604 2024-04-25T09:12:30.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:30.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:12:30.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-124554051604 2024-04-25T09:12:30.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:30.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051604 2024-04-25T09:12:30.594 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 124554051604 2024-04-25T09:12:30.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 124554051604' 2024-04-25T09:12:30.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:12:31.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051605 -lt 124554051604 2024-04-25T09:12:31.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:12:31.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T09:12:31.428 INFO:tasks.workunit.client.0.smithi134.stderr:dumped all 2024-04-25T09:12:31.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:127: rados_get_data: COUNT=1 2024-04-25T09:12:31.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:128: rados_get_data: test 1 = 1 2024-04-25T09:12:31.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: get_osds pool-rep obj-eio-390086 2024-04-25T09:12:31.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:12:31.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-390086 2024-04-25T09:12:31.446 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:31.446 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:12:31.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:12:31.844 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:31.844 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:12:31.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:12:31.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: object_osds=('3' '1' '0') 2024-04-25T09:12:31.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:130: rados_get_data: local object_osds 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:131: rados_get_data: local primary=3 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:132: rados_get_data: local bad_peer=1 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:133: rados_get_data: inject_eio rep data pool-rep obj-eio-390086 td/osd-rep-recov-eio.sh 0 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-390086 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:31.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:12:31.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:12:31.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:12:31.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-390086 2024-04-25T09:12:31.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:12:31.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-390086 2024-04-25T09:12:31.846 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:12:31.846 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:32.273 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:12:32.273 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:32.274 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:12:32.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:12:32.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:12:32.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:12:32.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:12:32.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:12:32.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:12:32.275 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:12:32.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:12:32.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:12:32.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:12:32.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:12:32.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:12:32.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:12:32.276 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:12:32.277 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:12:32.277 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:12:32.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:12:32.277 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:32.277 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:32.277 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:32.277 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:12:32.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:12:32.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:12:32.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:12:32.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:12:32.393 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:12:32.394 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:12:32.394 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:12:32.394 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:32.394 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:32.394 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:32.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:12:32.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:12:32.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-eio-390086 2024-04-25T09:12:32.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:134: rados_get_data: inject_eio rep data pool-rep obj-eio-390086 td/osd-rep-recov-eio.sh 1 2024-04-25T09:12:32.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:12:32.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:12:32.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:12:32.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:12:32.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:12:32.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:12:32.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-390086 2024-04-25T09:12:32.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:12:32.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:32.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:12:32.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:12:32.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:12:32.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-390086 2024-04-25T09:12:32.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:12:32.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-390086 2024-04-25T09:12:32.515 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:32.515 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:12:32.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:12:32.913 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:32.913 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:12:32.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:12:32.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:12:32.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:12:32.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:12:32.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:12:32.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:12:32.914 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:12:32.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:12:32.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:12:32.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:12:32.916 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:12:33.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:12:33.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:12:33.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:12:33.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:12:33.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:12:33.038 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:12:33.038 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:33.038 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:33.038 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:33.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:12:33.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:12:33.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-eio-390086 2024-04-25T09:12:33.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:136: rados_get_data: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:12:33.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:12:33.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:12:33.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:12:33.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:12:33.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:12:33.159 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:12:33.159 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:12:33.159 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:12:33.160 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:33.160 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:33.160 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:33.160 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:12:33.160 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:12:33.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:12:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:137: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-390086 2024-04-25T09:12:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:12:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-390086 2024-04-25T09:12:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:12:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:12:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:12:33.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-390086 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:33.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:33.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:33.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:140: rados_get_data: wait_for_clean 2024-04-25T09:12:33.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:12:33.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:12:33.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:12:33.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:12:33.301 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:12:33.301 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:12:33.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:12:33.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:12:33.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:12:33.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:12:33.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:12:33.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:12:33.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:12:33.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:12:33.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:12:33.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:12:33.773 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:33.773 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:12:33.774 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:12:33.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:12:33.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:33.774 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:12:33.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803823 2024-04-25T09:12:33.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803823 2024-04-25T09:12:33.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823' 2024-04-25T09:12:33.906 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:33.907 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:12:34.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574888 2024-04-25T09:12:34.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574888 2024-04-25T09:12:34.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-55834574888' 2024-04-25T09:12:34.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:34.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:12:34.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313246 2024-04-25T09:12:34.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313246 2024-04-25T09:12:34.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-55834574888 2-90194313246' 2024-04-25T09:12:34.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:34.181 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:12:34.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051607 2024-04-25T09:12:34.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051607 2024-04-25T09:12:34.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803823 1-55834574888 2-90194313246 3-124554051607' 2024-04-25T09:12:34.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:34.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803823 2024-04-25T09:12:34.329 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:34.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:12:34.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803823 2024-04-25T09:12:34.331 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:34.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803823 2024-04-25T09:12:34.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803823' 2024-04-25T09:12:34.332 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803823 2024-04-25T09:12:34.332 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:34.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803823 -lt 25769803823 2024-04-25T09:12:34.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:34.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574888 2024-04-25T09:12:34.888 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:34.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:12:34.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574888 2024-04-25T09:12:34.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:34.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574888 2024-04-25T09:12:34.892 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574888 2024-04-25T09:12:34.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574888' 2024-04-25T09:12:34.893 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:35.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574888 -lt 55834574888 2024-04-25T09:12:35.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:35.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313246 2024-04-25T09:12:35.304 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:35.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:12:35.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313246 2024-04-25T09:12:35.307 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:35.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313246 2024-04-25T09:12:35.308 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 90194313246 2024-04-25T09:12:35.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313246' 2024-04-25T09:12:35.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:12:35.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313246 -lt 90194313246 2024-04-25T09:12:35.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:35.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-124554051607 2024-04-25T09:12:35.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:35.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:12:35.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-124554051607 2024-04-25T09:12:35.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:35.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051607 2024-04-25T09:12:35.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 124554051607' 2024-04-25T09:12:35.751 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 124554051607 2024-04-25T09:12:35.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:12:36.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051607 -lt 124554051607 2024-04-25T09:12:36.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:12:36.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:36.169 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:36.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:12:36.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:12:36.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:12:36.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:12:36.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:12:36.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:12:36.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:12:36.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:12:37.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:12:37.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:12:37.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:37.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:37.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:12:37.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:12:37.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:142: rados_get_data: inject_eio rep data pool-rep obj-eio-390086 td/osd-rep-recov-eio.sh 0 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-390086 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:12:37.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:12:37.664 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-390086 2024-04-25T09:12:37.664 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:12:37.664 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-390086 2024-04-25T09:12:37.664 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:37.664 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:12:38.123 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:12:38.123 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:38.123 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:12:38.123 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:12:38.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:12:38.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:12:38.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:12:38.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:12:38.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:12:38.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:12:38.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:12:38.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:12:38.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:12:38.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:12:38.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:12:38.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:12:38.127 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:12:38.127 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:12:38.127 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:12:38.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:12:38.127 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:38.127 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:38.127 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:38.127 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:12:38.128 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:12:38.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:12:38.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:12:38.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:12:38.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:12:38.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:12:38.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:12:38.236 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:38.236 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:38.236 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:38.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:12:38.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:12:38.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-eio-390086 2024-04-25T09:12:38.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:143: rados_get_data: inject_eio rep data pool-rep obj-eio-390086 td/osd-rep-recov-eio.sh 2 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-390086 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=2 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-390086 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:12:38.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-390086 2024-04-25T09:12:38.355 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:38.355 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:12:38.778 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:12:38.778 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:38.778 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:12:38.778 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:12:38.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:12:38.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:12:38.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-04-25T09:12:38.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:12:38.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:12:38.779 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-04-25T09:12:38.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:12:38.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-25T09:12:38.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:12:38.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-25T09:12:38.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:12:38.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:12:38.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:12:38.781 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-25T09:12:38.781 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T09:12:38.781 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T09:12:38.781 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:38.782 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:38.782 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:38.782 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.0.asok 2024-04-25T09:12:38.782 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:12:38.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:12:38.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:12:38.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:12:38.907 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-04-25T09:12:38.907 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T09:12:38.907 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T09:12:38.908 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:38.908 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:38.908 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:38.908 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.0.asok 2024-04-25T09:12:38.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:12:38.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.0.asok injectdataerr pool-rep obj-eio-390086 2024-04-25T09:12:39.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:144: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-390086 2024-04-25T09:12:39.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:39.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:12:39.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-390086 2024-04-25T09:12:39.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:12:39.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:12:39.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:12:39.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-eio-390086 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:39.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:39.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:39.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:146: rados_get_data: wait_for_clean 2024-04-25T09:12:39.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:12:39.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:12:39.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:12:39.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:12:39.053 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:12:39.053 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:12:39.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:12:39.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:12:39.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:12:39.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:12:39.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:12:39.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:12:39.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:12:39.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:12:39.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:12:39.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:12:39.513 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:39.513 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:12:39.513 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:12:39.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:12:39.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:39.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:12:39.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803826 2024-04-25T09:12:39.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803826 2024-04-25T09:12:39.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826' 2024-04-25T09:12:39.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:39.703 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:12:39.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574892 2024-04-25T09:12:39.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574892 2024-04-25T09:12:39.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826 1-55834574892' 2024-04-25T09:12:39.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:39.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:12:40.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313249 2024-04-25T09:12:40.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313249 2024-04-25T09:12:40.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826 1-55834574892 2-90194313249' 2024-04-25T09:12:40.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:40.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:12:40.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051610 2024-04-25T09:12:40.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051610 2024-04-25T09:12:40.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803826 1-55834574892 2-90194313249 3-124554051610' 2024-04-25T09:12:40.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:40.143 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803826 2024-04-25T09:12:40.143 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:40.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:12:40.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803826 2024-04-25T09:12:40.145 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:40.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803826 2024-04-25T09:12:40.146 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803826 2024-04-25T09:12:40.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803826' 2024-04-25T09:12:40.147 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:40.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803826 -lt 25769803826 2024-04-25T09:12:40.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:40.549 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:40.549 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574892 2024-04-25T09:12:40.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:12:40.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574892 2024-04-25T09:12:40.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:40.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574892 2024-04-25T09:12:40.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574892' 2024-04-25T09:12:40.553 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574892 2024-04-25T09:12:40.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:40.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574892 -lt 55834574892 2024-04-25T09:12:40.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:40.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313249 2024-04-25T09:12:40.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:40.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:12:40.973 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313249 2024-04-25T09:12:40.973 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:40.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313249 2024-04-25T09:12:40.975 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 90194313249 2024-04-25T09:12:40.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313249' 2024-04-25T09:12:40.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:12:41.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313249 -lt 90194313249 2024-04-25T09:12:41.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:41.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-124554051610 2024-04-25T09:12:41.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:41.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:12:41.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-124554051610 2024-04-25T09:12:41.419 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:41.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051610 2024-04-25T09:12:41.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 124554051610' 2024-04-25T09:12:41.420 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 124554051610 2024-04-25T09:12:41.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:12:41.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051611 -lt 124554051610 2024-04-25T09:12:41.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:12:41.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:41.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:42.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:12:42.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:12:42.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:12:42.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:12:42.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:12:42.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:12:42.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:12:42.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:12:42.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:12:42.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:12:42.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:42.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:43.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:12:43.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:12:43.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:12:43.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: ceph pg 2.0 query 2024-04-25T09:12:43.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:12:43.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:147: rados_get_data: COUNT=3 2024-04-25T09:12:43.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:148: rados_get_data: test 3 = 3 2024-04-25T09:12:43.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:149: rados_get_data: flush_pg_stats 2024-04-25T09:12:43.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:12:43.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:12:43.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:12:43.903 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:43.903 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:12:43.903 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:12:43.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:12:43.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:43.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:12:44.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803829 2024-04-25T09:12:44.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803829 2024-04-25T09:12:44.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803829' 2024-04-25T09:12:44.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:44.050 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:12:44.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574894 2024-04-25T09:12:44.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574894 2024-04-25T09:12:44.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803829 1-55834574894' 2024-04-25T09:12:44.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:44.198 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:12:44.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313252 2024-04-25T09:12:44.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313252 2024-04-25T09:12:44.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803829 1-55834574894 2-90194313252' 2024-04-25T09:12:44.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:44.334 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:12:44.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051613 2024-04-25T09:12:44.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051613 2024-04-25T09:12:44.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803829 1-55834574894 2-90194313252 3-124554051613' 2024-04-25T09:12:44.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:44.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803829 2024-04-25T09:12:44.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:44.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:12:44.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803829 2024-04-25T09:12:44.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:44.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803829 2024-04-25T09:12:44.473 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803829 2024-04-25T09:12:44.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803829' 2024-04-25T09:12:44.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:44.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803829 -lt 25769803829 2024-04-25T09:12:44.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:44.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574894 2024-04-25T09:12:44.856 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:44.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:12:44.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574894 2024-04-25T09:12:44.858 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:44.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574894 2024-04-25T09:12:44.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574894' 2024-04-25T09:12:44.860 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574894 2024-04-25T09:12:44.860 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:45.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574894 -lt 55834574894 2024-04-25T09:12:45.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:45.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313252 2024-04-25T09:12:45.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:45.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:12:45.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313252 2024-04-25T09:12:45.267 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:45.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313252 2024-04-25T09:12:45.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313252' 2024-04-25T09:12:45.269 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 90194313252 2024-04-25T09:12:45.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:12:45.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313252 -lt 90194313252 2024-04-25T09:12:45.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:45.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-124554051613 2024-04-25T09:12:45.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:45.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:12:45.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-124554051613 2024-04-25T09:12:45.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:45.715 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 124554051613 2024-04-25T09:12:45.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051613 2024-04-25T09:12:45.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 124554051613' 2024-04-25T09:12:45.716 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:12:46.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051611 -lt 124554051613 2024-04-25T09:12:46.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:12:47.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:12:47.137 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:12:47.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051614 -lt 124554051613 2024-04-25T09:12:47.558 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T09:12:47.558 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:12:47.940 INFO:tasks.workunit.client.0.smithi134.stderr:dumped all 2024-04-25T09:12:47.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:150: rados_get_data: COUNT=4 2024-04-25T09:12:47.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:151: rados_get_data: test 4 = 4 2024-04-25T09:12:47.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:153: rados_get_data: inject_eio rep data pool-rep obj-eio-390086 td/osd-rep-recov-eio.sh 0 2024-04-25T09:12:47.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-390086 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-390086 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:12:47.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-390086 2024-04-25T09:12:47.961 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:47.961 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:12:48.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:12:48.354 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:48.354 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:12:48.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:12:48.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:12:48.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:12:48.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:12:48.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:12:48.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:12:48.355 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:12:48.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:12:48.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:12:48.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:12:48.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:12:48.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:12:48.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:12:48.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:12:48.357 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:12:48.357 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:12:48.357 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:12:48.357 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:48.357 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:48.358 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:48.358 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:12:48.358 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:12:48.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:12:48.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:12:48.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:12:48.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:12:48.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:12:48.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:12:48.470 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:48.470 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:48.470 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:48.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:12:48.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:12:48.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-eio-390086 2024-04-25T09:12:48.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:154: rados_get_data: inject_eio rep data pool-rep obj-eio-390086 td/osd-rep-recov-eio.sh 1 2024-04-25T09:12:48.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:12:48.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-390086 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:12:48.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:12:48.588 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-390086 2024-04-25T09:12:48.588 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:12:48.588 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-390086 2024-04-25T09:12:48.588 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:48.588 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:12:49.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:12:49.031 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:49.031 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:12:49.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:12:49.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:12:49.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:12:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:12:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:12:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:12:49.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:12:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:12:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:12:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:12:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:12:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:12:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:12:49.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:12:49.034 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:12:49.034 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:12:49.035 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:12:49.035 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:49.035 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:49.035 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:49.035 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:12:49.035 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:12:49.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:12:49.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:12:49.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:12:49.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:12:49.150 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:12:49.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:12:49.150 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:49.150 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:49.151 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:49.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:12:49.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:12:49.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-eio-390086 2024-04-25T09:12:49.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:155: rados_get_data: inject_eio rep data pool-rep obj-eio-390086 td/osd-rep-recov-eio.sh 2 2024-04-25T09:12:49.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:12:49.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-eio-390086 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=2 2024-04-25T09:12:49.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:12:49.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-eio-390086 2024-04-25T09:12:49.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:12:49.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-eio-390086 2024-04-25T09:12:49.262 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-eio-390086 2024-04-25T09:12:49.262 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:12:49.654 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:12:49.654 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:49.654 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:12:49.654 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:12:49.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:12:49.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:12:49.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=0 2024-04-25T09:12:49.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:12:49.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:12:49.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/0/type 2024-04-25T09:12:49.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:12:49.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 0 bluestore_debug_inject_read_err true 2024-04-25T09:12:49.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:12:49.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=0 2024-04-25T09:12:49.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:12:49.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:12:49.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:12:49.656 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.0 2024-04-25T09:12:49.657 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T09:12:49.657 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T09:12:49.657 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:49.657 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:49.657 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:49.658 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.0.asok 2024-04-25T09:12:49.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.0.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:12:49.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:12:49.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:12:49.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:12:49.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.0 2024-04-25T09:12:49.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-04-25T09:12:49.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-04-25T09:12:49.772 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:12:49.772 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:12:49.772 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:12:49.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.0.asok 2024-04-25T09:12:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:12:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.0.asok injectdataerr pool-rep obj-eio-390086 2024-04-25T09:12:49.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:156: rados_get_data: rados_get td/osd-rep-recov-eio.sh pool-rep obj-eio-390086 hang 2024-04-25T09:12:49.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:12:49.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:12:49.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-eio-390086 2024-04-25T09:12:49.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=hang 2024-04-25T09:12:49.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' hang = fail ']' 2024-04-25T09:12:49.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' hang = hang ']' 2024-04-25T09:12:49.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:98: rados_get: timeout 5 rados --pool pool-rep get obj-eio-390086 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:12:54.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:99: rados_get: test 124 = 124 2024-04-25T09:12:54.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:100: rados_get: return 2024-04-25T09:12:54.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:158: rados_get_data: wait_for_clean 2024-04-25T09:12:54.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:12:54.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:12:54.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:12:54.896 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:12:54.896 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:12:54.896 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:12:54.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:12:54.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:12:54.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:12:55.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:12:55.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:12:55.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:12:55.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:12:55.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:12:55.017 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:12:55.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:12:55.437 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:12:55.438 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:12:55.438 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:12:55.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:12:55.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:55.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:12:55.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803833 2024-04-25T09:12:55.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803833 2024-04-25T09:12:55.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803833' 2024-04-25T09:12:55.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:55.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:12:55.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574899 2024-04-25T09:12:55.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574899 2024-04-25T09:12:55.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803833 1-55834574899' 2024-04-25T09:12:55.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:55.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:12:55.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313257 2024-04-25T09:12:55.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313257 2024-04-25T09:12:55.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803833 1-55834574899 2-90194313257' 2024-04-25T09:12:55.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:12:55.905 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:12:56.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051618 2024-04-25T09:12:56.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051618 2024-04-25T09:12:56.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803833 1-55834574899 2-90194313257 3-124554051618' 2024-04-25T09:12:56.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:56.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803833 2024-04-25T09:12:56.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:56.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:12:56.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803833 2024-04-25T09:12:56.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:56.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803833 2024-04-25T09:12:56.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803833' 2024-04-25T09:12:56.048 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803833 2024-04-25T09:12:56.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:56.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803831 -lt 25769803833 2024-04-25T09:12:56.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:12:57.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:12:57.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:12:57.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803834 -lt 25769803833 2024-04-25T09:12:57.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:57.864 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:57.864 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574899 2024-04-25T09:12:57.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:12:57.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574899 2024-04-25T09:12:57.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:57.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574899 2024-04-25T09:12:57.867 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574899 2024-04-25T09:12:57.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574899' 2024-04-25T09:12:57.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:12:58.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574899 -lt 55834574899 2024-04-25T09:12:58.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:58.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313257 2024-04-25T09:12:58.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:58.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:12:58.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313257 2024-04-25T09:12:58.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:58.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313257 2024-04-25T09:12:58.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313257' 2024-04-25T09:12:58.303 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 90194313257 2024-04-25T09:12:58.303 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:12:58.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313257 -lt 90194313257 2024-04-25T09:12:58.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:12:58.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-124554051618 2024-04-25T09:12:58.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:12:58.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:12:58.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-124554051618 2024-04-25T09:12:58.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:12:58.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051618 2024-04-25T09:12:58.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 124554051618' 2024-04-25T09:12:58.799 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 124554051618 2024-04-25T09:12:58.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:12:59.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051618 -lt 124554051618 2024-04-25T09:12:59.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:12:59.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:12:59.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:12:59.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:12:59.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:12:59.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:12:59.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:12:59.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:12:59.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:12:59.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:12:59.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:00.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:00.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:00.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:00.168 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:00.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:00.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-04-25T09:13:00.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:13:00.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-04-25T09:13:00.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:00.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:13:00.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:00.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:00.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:00.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:00.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:00.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:00.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:00.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:01.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:01.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:01.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:01.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:01.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:01.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:13:01.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:01.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:01.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:01.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:01.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:01.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:01.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:02.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:02.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:02.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T09:13:02.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:02.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T09:13:02.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:02.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:02.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:02.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:02.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:02.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:02.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:02.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:02.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:02.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:02.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:02.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:03.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:03.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:13:03.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:03.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:03.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:03.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:03.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:03.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:03.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:04.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:04.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:04.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 13 )) 2024-04-25T09:13:04.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:04.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-04-25T09:13:04.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:04.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:04.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:04.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:04.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:04.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:04.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:04.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:04.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:04.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:04.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:04.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:05.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:05.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:13:05.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:05.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:05.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:05.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:05.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:05.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:05.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:06.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:06.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:06.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 13 )) 2024-04-25T09:13:06.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:06.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.8 2024-04-25T09:13:06.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:06.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:06.828 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:06.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:06.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:06.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:06.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:06.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:07.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:07.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:07.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:07.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:07.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:07.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:13:07.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:07.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:07.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:07.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:07.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:07.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:07.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:08.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:08.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:08.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 13 )) 2024-04-25T09:13:08.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:08.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 1.6 2024-04-25T09:13:09.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:09.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:09.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:09.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:09.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:09.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:09.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:09.912 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:10.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:10.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:10.323 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:10.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:10.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:10.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:13:10.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:10.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:10.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:10.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:10.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:10.872 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:10.872 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:11.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:11.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:11.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 5 >= 13 )) 2024-04-25T09:13:11.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:11.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 3.2 2024-04-25T09:13:14.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:14.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:14.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:14.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:14.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:14.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:14.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:14.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:15.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:15.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:15.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:15.015 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:15.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:15.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:13:15.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:15.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:15.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:15.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:15.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:15.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:15.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:16.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:16.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:16.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 6 >= 13 )) 2024-04-25T09:13:16.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:16.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 6.4 2024-04-25T09:13:22.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:22.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:22.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:22.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:22.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:22.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:22.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:22.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:22.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:22.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:22.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:22.895 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:23.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:23.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:13:23.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:23.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:23.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:23.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:23.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:23.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:23.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:23.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:23.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:23.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 7 >= 13 )) 2024-04-25T09:13:23.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:23.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 12.8 2024-04-25T09:13:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:36.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:36.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:36.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:36.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:36.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:36.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:36.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:37.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:37.279 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:37.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:37.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:37.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:37.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:13:37.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:37.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:37.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:37.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:37.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:37.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:37.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:38.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:38.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:38.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 8 >= 13 )) 2024-04-25T09:13:38.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:38.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:13:53.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:13:53.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:13:53.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:13:53.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:13:53.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:13:53.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:13:53.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:13:53.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:13:53.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:13:53.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:13:53.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:13:53.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:13:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:13:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:13:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:13:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:13:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:13:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:13:54.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:13:54.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:13:54.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:13:54.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:13:54.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:13:54.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 9 >= 13 )) 2024-04-25T09:13:54.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:13:54.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:14:09.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:14:09.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:09.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:09.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:09.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:09.842 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:09.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:09.843 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:10.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:14:10.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:10.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:10.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:10.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:14:10.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:14:10.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:14:10.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:14:10.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:14:10.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:14:10.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:14:10.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:14:10.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:14:11.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:14:11.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:14:11.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 10 >= 13 )) 2024-04-25T09:14:11.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:14:11.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:14:26.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:14:26.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:26.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:26.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:26.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:26.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:26.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:26.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:26.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:14:26.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:26.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:26.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:14:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:14:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:14:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:14:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:14:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:14:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:14:27.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:14:27.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:14:27.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:14:27.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:14:27.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 11 >= 13 )) 2024-04-25T09:14:27.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:14:27.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 15 2024-04-25T09:14:42.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:14:42.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:42.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:42.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:42.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:42.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:42.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:42.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:43.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:14:43.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:43.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:43.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:43.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:14:43.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:14:43.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:14:43.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:14:43.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:14:43.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:14:43.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:14:43.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:14:43.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:14:44.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:14:44.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:14:44.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 12 >= 13 )) 2024-04-25T09:14:44.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:14:44.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 4.5 2024-04-25T09:14:48.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:14:48.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:14:48.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:14:48.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:14:48.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:14:48.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:14:48.769 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:14:48.770 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:14:49.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:14:49.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:14:49.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:14:49.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:14:49.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:14:49.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:14:49.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:14:49.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:14:49.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:14:49.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:14:49.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:14:49.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:14:49.716 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:14:50.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:14:50.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:14:50.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 13 >= 13 )) 2024-04-25T09:14:50.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-04-25T09:14:50.873 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T09:14:50.873 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_fingerprint": "8631cadb-d67a-48b7-8207-da3f5c6a00dd", 2024-04-25T09:14:50.873 INFO:tasks.workunit.client.0.smithi134.stdout: "version": "19.0.0-3227-gc37f63b3", 2024-04-25T09:14:50.873 INFO:tasks.workunit.client.0.smithi134.stdout: "commit": "c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f", 2024-04-25T09:14:50.873 INFO:tasks.workunit.client.0.smithi134.stdout: "timestamp": "2024-04-25T09:14:50.883947+0000", 2024-04-25T09:14:50.873 INFO:tasks.workunit.client.0.smithi134.stdout: "tag": "", 2024-04-25T09:14:50.873 INFO:tasks.workunit.client.0.smithi134.stdout: "health": { 2024-04-25T09:14:50.873 INFO:tasks.workunit.client.0.smithi134.stdout: "status": "HEALTH_ERR", 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: "checks": { 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: "OBJECT_UNFOUND": { 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1/1 objects unfound (100.000%)", 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 2.0 has 1 unfound objects" 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T09:14:50.874 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "PG_DAMAGED": { 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_ERR", 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "Possible data damage: 1 pg recovery_unfound", 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "PG_DEGRADED": { 2024-04-25T09:14:50.875 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "Degraded data redundancy: 3/3 objects degraded (100.000%), 1 pg degraded", 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "pg 2.0 is active+recovery_unfound+degraded, acting [3,1,0], 1 unfound" 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: "POOL_APP_NOT_ENABLED": { 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:14:50.876 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "2 pool(s) do not have an application enabled", 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 2 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [ 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "application not enabled on pool 'foo'" 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "application not enabled on pool 'pool-rep'" 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.877 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: "SLOW_OPS": { 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: "severity": "HEALTH_WARN", 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: "summary": { 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: "message": "1 slow ops, oldest one blocked for 119 sec, osd.3 has slow ops", 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: "count": 1 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: "detail": [], 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: "muted": false 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.878 INFO:tasks.workunit.client.0.smithi134.stdout: "mutes": [] 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_first_committed": 1, 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap_last_committed": 1, 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "monmap": { 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "52ea6452-1094-49d7-af80-9f2c0397539e", 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T09:09:27.971047Z", 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T09:09:27.971047Z", 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release": 19, 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "min_mon_release_name": "squid", 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "election_strategy": 1, 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "disallowed_leaders: ": "", 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": false, 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "tiebreaker_mon": "", 2024-04-25T09:14:50.879 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_ranks: ": "", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "features": { 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "persistent": [ 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "kraken", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "luminous", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "mimic", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap-prune", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "nautilus", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "octopus", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "pacific", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "elector-pinging", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "quincy", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "reef", 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "squid" 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.880 INFO:tasks.workunit.client.0.smithi134.stdout: "optional": [] 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: "mons": [ 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: "rank": 0, 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "a", 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7140", 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 0 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.881 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:7140/0", 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:7140/0", 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: "priority": 0, 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 0, 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_location": "{}" 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: "quorum": [ 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: 0 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap": { 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 36, 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: "fsid": "52ea6452-1094-49d7-af80-9f2c0397539e", 2024-04-25T09:14:50.882 INFO:tasks.workunit.client.0.smithi134.stdout: "created": "2024-04-25T09:09:28.999863+0000", 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T09:12:14.911028+0000", 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "last_up_change": "2024-04-25T09:12:04.845574+0000", 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "last_in_change": "2024-04-25T09:11:26.324648+0000", 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_num": 5799936, 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_set": [ 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "pglog_hardlimit", 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snapdirs", 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "recovery_deletes", 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "sortbitwise" 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_version": 9, 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "full_ratio": 0.99000000953674316, 2024-04-25T09:14:50.883 INFO:tasks.workunit.client.0.smithi134.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "nearfull_ratio": 0.85000002384185791, 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_snapshot": "", 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_max": 2, 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "max_osd": 4, 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "require_min_compat_client": "luminous", 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "min_compat_client": "jewel", 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "require_osd_release": "squid", 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "allow_crimson": false, 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "pools": [ 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_name": "foo", 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "create_time": "2024-04-25T09:09:30.047157+0000", 2024-04-25T09:14:50.884 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": 1, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_names": "hashpspool", 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 3, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "min_size": 2, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_rule": 0, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "object_hash": 2, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_autoscale_mode": "off", 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num": 8, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num": 8, 2024-04-25T09:14:50.885 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num_target": 8, 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_target": 8, 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_pending": 8, 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pg_merge_meta": { 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "source_pgid": "0.0", 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "ready_epoch": 0, 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_started": 0, 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": 0, 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "source_version": "0'0", 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "target_version": "0'0" 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "last_change": "8", 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend": "0", 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T09:14:50.886 INFO:tasks.workunit.client.0.smithi134.stdout: "auid": 0, 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_mode": "selfmanaged", 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_seq": 0, 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_epoch": 0, 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_snaps": [], 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps": "[]", 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_bytes": 0, 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_objects": 0, 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "tiers": [], 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "tier_of": -1, 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "read_tier": -1, 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "write_tier": -1, 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_mode": "none", 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_bytes": 0, 2024-04-25T09:14:50.887 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_objects": 0, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_flush_age": 0, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_evict_age": 0, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profile": "", 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_params": { 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "none" 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_period": 0, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_count": 0, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "use_gmt_hitset": true, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-04-25T09:14:50.888 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_search_last_n": 0, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "grade_table": [], 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "stripe_width": 0, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "expected_num_objects": 0, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "fast_read": false, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "options": {}, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "application_metadata": {}, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "read_balance": { 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "score_type": "Fair distribution", 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "score_acting": 1.5, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "score_stable": 1.5, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_score": 1, 2024-04-25T09:14:50.889 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_acting": 1.5, 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_stable": 1.5, 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity_weighted": 1, 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity": 1, 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 2, 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_name": "pool-rep", 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "create_time": "2024-04-25T09:12:13.621958+0000", 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "flags": 1, 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "flags_names": "hashpspool", 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T09:14:50.890 INFO:tasks.workunit.client.0.smithi134.stdout: "size": 3, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "min_size": 2, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_rule": 0, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_count": 0, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_target": 0, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_barrier": 0, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "object_hash": 2, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_autoscale_mode": "off", 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num": 1, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num": 1, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_placement_num_target": 1, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_target": 1, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_num_pending": 1, 2024-04-25T09:14:50.891 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pg_merge_meta": { 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "source_pgid": "0.0", 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "ready_epoch": 0, 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_started": 0, 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": 0, 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "source_version": "0'0", 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "target_version": "0'0" 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "last_change": "36", 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend": "0", 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_prenautilus": "0", 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "last_force_op_resend_preluminous": "0", 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "auid": 0, 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_mode": "selfmanaged", 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_seq": 0, 2024-04-25T09:14:50.892 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_epoch": 0, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_snaps": [], 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps": "[]", 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_bytes": 0, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "quota_max_objects": 0, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "tiers": [], 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "tier_of": -1, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "read_tier": -1, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "write_tier": -1, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_mode": "none", 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_bytes": 0, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "target_max_objects": 0, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-04-25T09:14:50.893 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_target_full_ratio_micro": 800000, 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_flush_age": 0, 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "cache_min_evict_age": 0, 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profile": "", 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_params": { 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "none" 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_period": 0, 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_count": 0, 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "use_gmt_hitset": true, 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "min_read_recency_for_promote": 0, 2024-04-25T09:14:50.894 INFO:tasks.workunit.client.0.smithi134.stdout: "min_write_recency_for_promote": 0, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_grade_decay_rate": 0, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "hit_set_search_last_n": 0, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "grade_table": [], 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "stripe_width": 0, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "expected_num_objects": 0, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "fast_read": false, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "options": {}, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "application_metadata": {}, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "read_balance": { 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "score_type": "Fair distribution", 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "score_acting": 4, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "score_stable": 4, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_score": 0.75, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_acting": 3, 2024-04-25T09:14:50.895 INFO:tasks.workunit.client.0.smithi134.stdout: "raw_score_stable": 3, 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity_weighted": 1, 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity": 1, 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "average_primary_affinity_weighted": 1 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "osds": [ 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "8acd0bf9-25ea-4404-92f6-ff02cb2fe45f", 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T09:14:50.896 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 6, 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 31, 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 0, 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6802", 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 538825488 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.897 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6803", 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 538825488 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6804", 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 538825488 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.898 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6805", 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 538825488 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6808", 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 538825488 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.899 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6809", 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 538825488 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6806", 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 538825488 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.900 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6807", 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 538825488 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6803/538825488", 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6805/538825488", 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6809/538825488", 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6807/538825488", 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.901 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "31dbc289-cf12-46bf-ba7d-2cd1678d7921", 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 13, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 31, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 0, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T09:14:50.902 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6810", 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 539324914 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6811", 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 539324914 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.903 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6812", 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 539324914 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6813", 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 539324914 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.904 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6816", 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 539324914 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6817", 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 539324914 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6814", 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 539324914 2024-04-25T09:14:50.905 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6815", 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 539324914 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6811/539324914", 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6813/539324914", 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6817/539324914", 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6815/539324914", 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T09:14:50.906 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "6975a61d-624b-4dd1-9841-7a4517d296d9", 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 21, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 31, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 0, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T09:14:50.907 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6818", 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1054361274 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6819", 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1054361274 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.908 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6820", 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1054361274 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6821", 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1054361274 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.909 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6824", 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1054361274 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6825", 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1054361274 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.910 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6822", 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1054361274 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6823", 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 1054361274 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6819/1054361274", 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6821/1054361274", 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6825/1054361274", 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6823/1054361274", 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T09:14:50.911 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 3, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "uuid": "bb0aeb49-2ea1-42ed-9ba3-76e332478323", 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 1, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "in": 1, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 1, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_affinity": 1, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_begin": 0, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "last_clean_end": 0, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 29, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "up_thru": 34, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "down_at": 0, 2024-04-25T09:14:50.912 INFO:tasks.workunit.client.0.smithi134.stdout: "lost_at": 0, 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addrs": { 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6826", 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3701627663 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6827", 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3701627663 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.913 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addrs": { 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6828", 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3701627663 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6829", 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3701627663 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addrs": { 2024-04-25T09:14:50.914 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6832", 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3701627663 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6833", 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3701627663 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addrs": { 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: "addrvec": [ 2024-04-25T09:14:50.915 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v2", 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6830", 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3701627663 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "v1", 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "addr": "127.0.0.1:6831", 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "nonce": 3701627663 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "public_addr": "127.0.0.1:6827/3701627663", 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "cluster_addr": "127.0.0.1:6829/3701627663", 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_back_addr": "127.0.0.1:6833/3701627663", 2024-04-25T09:14:50.916 INFO:tasks.workunit.client.0.smithi134.stdout: "heartbeat_front_addr": "127.0.0.1:6831/3701627663", 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "state": [ 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "exists", 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "up" 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_xinfo": [ 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "0.000000", 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T09:14:50.917 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T09:10:05.358963+0000", 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "0.000000", 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T09:10:43.150737+0000", 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.918 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "0.000000", 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T09:11:23.877438+0000", 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 3, 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "down_stamp": "0.000000", 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_probability": 0, 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "laggy_interval": 0, 2024-04-25T09:14:50.919 INFO:tasks.workunit.client.0.smithi134.stdout: "features": 4540701547738038271, 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "old_weight": 0, 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "last_purged_snaps_scrub": "2024-04-25T09:12:04.365515+0000", 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "dead_epoch": 0 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap": [], 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_items": [], 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_upmap_primaries": [], 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "pg_temp": [], 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "primary_temp": [], 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "blocklist": {}, 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "range_blocklist": {}, 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "erasure_code_profiles": { 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "default": { 2024-04-25T09:14:50.920 INFO:tasks.workunit.client.0.smithi134.stdout: "k": "2", 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "m": "2", 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "plugin": "jerasure", 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "technique": "reed_sol_van" 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "removed_snaps_queue": [], 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "new_removed_snaps": [], 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "new_purged_snaps": [], 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "crush_node_flags": {}, 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "device_class_flags": {}, 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode": { 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_enabled": false, 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_bucket_count": 0, 2024-04-25T09:14:50.921 INFO:tasks.workunit.client.0.smithi134.stdout: "degraded_stretch_mode": 0, 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "recovering_stretch_mode": 0, 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "stretch_mode_bucket": 0 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_metadata": [ 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6804/538825488,v1:127.0.0.1:6805/538825488]", 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:14:50.922 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/0/block", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T09:14:50.923 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:14:50.924 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:14:50.924 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:14:50.924 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T09:14:50.925 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T09:09:36.225685Z", 2024-04-25T09:14:50.925 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T09:14:50.925 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T09:14:50.925 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:14:50.925 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T09:14:50.925 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6802/538825488,v1:127.0.0.1:6803/538825488]", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/538825488,v1:127.0.0.1:6809/538825488]", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/538825488,v1:127.0.0.1:6807/538825488]", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:14:50.926 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/osd-rep-recov-eio.sh/0", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6812/539324914,v1:127.0.0.1:6813/539324914]", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:14:50.927 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/1/block", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:14:50.928 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T09:10:13.790624Z", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6810/539324914,v1:127.0.0.1:6811/539324914]", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/539324914,v1:127.0.0.1:6817/539324914]", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/539324914,v1:127.0.0.1:6815/539324914]", 2024-04-25T09:14:50.929 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/osd-rep-recov-eio.sh/1", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.930 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6820/1054361274,v1:127.0.0.1:6821/1054361274]", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:14:50.931 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/2/block", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T09:10:51.441709Z", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T09:14:50.932 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6818/1054361274,v1:127.0.0.1:6819/1054361274]", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6824/1054361274,v1:127.0.0.1:6825/1054361274]", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6822/1054361274,v1:127.0.0.1:6823/1054361274]", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:14:50.933 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/osd-rep-recov-eio.sh/2", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 3, 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "arch": "x86_64", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "back_addr": "[v2:127.0.0.1:6828/3701627663,v1:127.0.0.1:6829/3701627663]", 2024-04-25T09:14:50.934 INFO:tasks.workunit.client.0.smithi134.stdout: "back_iface": "", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs": "1", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_db": "0", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_dedicated_wal": "0", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluefs_single_shared_device": "1", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_allocation_from_file": "0", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_access_mode": "file", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_block_size": "4096", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_devices": "sda", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_path": "td/osd-rep-recov-eio.sh/3/block", 2024-04-25T09:14:50.935 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_rotational": "1", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_size": "107374182400", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_support_discard": "0", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_bdev_type": "hdd", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "bluestore_min_alloc_size": "4096", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_release": "squid", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_short": "19.0.0-3227-gc37f63b3", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "ceph_version_when_created": "ceph version 19.0.0-3227-gc37f63b3 (c37f63b3705f40f1bfcd7d8aeb311499c65cfa1f) squid (dev)", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "created_at": "2024-04-25T09:11:33.170204Z", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "default_device_class": "hdd", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNGV", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": "sda", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "distro": "ubuntu", 2024-04-25T09:14:50.936 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "distro_version": "22.04", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "front_addr": "[v2:127.0.0.1:6826/3701627663,v1:127.0.0.1:6827/3701627663]", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "front_iface": "", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_back_addr": "[v2:127.0.0.1:6832/3701627663,v1:127.0.0.1:6833/3701627663]", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_front_addr": "[v2:127.0.0.1:6830/3701627663,v1:127.0.0.1:6831/3701627663]", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "hostname": "smithi134", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "journal_rotational": "1", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_description": "#115-Ubuntu SMP Mon Apr 15 09:52:04 UTC 2024", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "kernel_version": "5.15.0-105-generic", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_swap_kb": "2097148", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "mem_total_kb": "32746884", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "objectstore_numa_unknown_devices": "sda", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "os": "Linux", 2024-04-25T09:14:50.937 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_data": "td/osd-rep-recov-eio.sh/3", 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_objectstore": "bluestore", 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: "osdspec_affinity": "", 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: "rotational": "1" 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_clean_epochs": { 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: "min_last_epoch_clean": 0, 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: "last_epoch_clean": { 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: "per_pool": [ 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: "floor": 4294967295 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.938 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_epochs": [ 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 8 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 14 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 21 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.939 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 3, 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 29 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_first_committed": 1, 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "osdmap_last_committed": 36, 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "crushmap": { 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "devices": [ 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.0", 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T09:14:50.940 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.1", 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.2", 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 3, 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd.3", 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: "class": "hdd" 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.941 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: "types": [ 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 0, 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "osd" 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "host" 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 2, 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "chassis" 2024-04-25T09:14:50.942 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 3, 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "rack" 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 4, 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "row" 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 5, 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pdu" 2024-04-25T09:14:50.943 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 6, 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "pod" 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 7, 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "room" 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 8, 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "datacenter" 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.944 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 9, 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "zone" 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 10, 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "region" 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "root" 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: "buckets": [ 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.945 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -1, 2024-04-25T09:14:50.946 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default", 2024-04-25T09:14:50.946 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T09:14:50.946 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T09:14:50.946 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 25608, 2024-04-25T09:14:50.946 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T09:14:50.946 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T09:14:50.946 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T09:14:50.947 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.947 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T09:14:50.947 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 25608, 2024-04-25T09:14:50.947 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -2, 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "default~hdd", 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 11, 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "root", 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 25608, 2024-04-25T09:14:50.948 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 25608, 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -3, 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134", 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T09:14:50.949 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 25608, 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T09:14:50.950 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 3, 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 3 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.951 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "id": -4, 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "name": "smithi134~hdd", 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "type_id": 1, 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "type_name": "host", 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 25608, 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "alg": "straw2", 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "hash": "rjenkins1", 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "items": [ 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 0, 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 0 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.952 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 1, 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 1 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 2, 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 2 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.953 INFO:tasks.workunit.client.0.smithi134.stdout: "id": 3, 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: "weight": 6402, 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: "pos": 3 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: "rules": [ 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_id": 0, 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: "rule_name": "replicated_rule", 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: "type": 1, 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: "steps": [ 2024-04-25T09:14:50.954 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "take", 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: "item": -1, 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: "item_name": "default" 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "choose_firstn", 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 0, 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: "type": "osd" 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: "op": "emit" 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: ] 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.955 INFO:tasks.workunit.client.0.smithi134.stdout: "tunables": { 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_tries": 0, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_local_fallback_tries": 0, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_total_tries": 50, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_descend_once": 1, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_vary_r": 1, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "chooseleaf_stable": 1, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_descents": 100, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "msr_collision_tries": 100, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "straw_calc_version": 1, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "allowed_bucket_algs": 54, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "profile": "jewel", 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "optimal_tunables": 1, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "legacy_tunables": 0, 2024-04-25T09:14:50.956 INFO:tasks.workunit.client.0.smithi134.stdout: "minimum_required_version": "jewel", 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables": 1, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables2": 1, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v2_rules": 0, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables3": 1, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v3_rules": 0, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v4_buckets": 1, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "require_feature_tunables5": 1, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "has_v5_rules": 0, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "has_msr_rules": 0 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "choose_args": {} 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "fsmap": { 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 1, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "default_fscid": -1, 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": { 2024-04-25T09:14:50.957 INFO:tasks.workunit.client.0.smithi134.stdout: "compat": {}, 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "ro_compat": {}, 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "incompat": { 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_1": "base v0.20", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_2": "client writeable ranges", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_3": "default file layouts on dirs", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_4": "dir inode in separate object", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_5": "mds uses versioned encoding", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_6": "dirfrag is stored in omap", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_8": "no anchor table", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_9": "file layout v2", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_10": "snaprealm v2", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_11": "minor log segments", 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_12": "quiesce subvolumes" 2024-04-25T09:14:50.958 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "feature_flags": { 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "enable_multiple": true, 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "ever_enabled_multiple": true 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "standbys": [], 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "filesystems": [] 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_first_committed": 1, 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "mdsmap_last_committed": 1, 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "auth": { 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 1, 2024-04-25T09:14:50.959 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 6, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_secrets": 4 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 9, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_active": 9, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_unknown": 0, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osd": 4, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_sum": { 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 4096, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 1, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 3, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 1, 2024-04-25T09:14:50.960 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 3, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 1, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 1, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 3, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 12, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 1, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 4, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 3, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 12288, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T09:14:50.961 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T09:14:50.962 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 3 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T09:14:50.963 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 1, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 1, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 27, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 27, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 0 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum": { 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 27, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 4, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 4, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 4, 2024-04-25T09:14:50.964 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 419430400, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 108504, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 1196, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 6, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 107257, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 419321896, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 429496729600, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 429385621504, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 1224704, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 405100, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T09:14:50.965 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 6378, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 109831958 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 4, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [ 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.966 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: 1 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 262144 2024-04-25T09:14:50.967 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_sum_by_class": { 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "hdd": { 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "up_from": 0, 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 0, 2024-04-25T09:14:50.968 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pgs": 27, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "num_osds": 4, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_osds": 4, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "num_per_pool_omap_osds": 4, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "kb": 419430400, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used": 108504, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_data": 1196, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_omap": 6, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_used_meta": 107257, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "kb_avail": 419321896, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "statfs": { 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 429496729600, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 429385621504, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 1224704, 2024-04-25T09:14:50.969 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 405100, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 6378, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 109831958 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "hb_peers": [], 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "snap_trim_queue_len": 0, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "num_snap_trimming": 0, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shards_repaired": 4, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "op_queue_age_hist": { 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: "histogram": [ 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.970 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.971 INFO:tasks.workunit.client.0.smithi134.stdout: 0, 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: 1 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: "upper_bound": 262144 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: "perf_stat": { 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ms": 0, 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ms": 0, 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: "commit_latency_ns": 0, 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: "apply_latency_ns": 0 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: "alerts": [], 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: "network_ping_times": [] 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: "pool_stats": [ 2024-04-25T09:14:50.972 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 2, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 1, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 4096, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 1, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 3, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 1, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 3, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 1, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 1, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 3, 2024-04-25T09:14:50.973 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 12, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 1, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 4, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 3, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 12288, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T09:14:50.974 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T09:14:50.975 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 3 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 12288, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 12288, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T09:14:50.976 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 1, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 1, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 3, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 3, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 3 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "poolid": 1, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg": 8, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "stat_sum": { 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes": 0, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects": 0, 2024-04-25T09:14:50.977 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_clones": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_object_copies": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing_on_primary": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_missing": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_degraded": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_misplaced": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_unfound": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_dirty": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_whiteouts": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_read_kb": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_write_kb": 0, 2024-04-25T09:14:50.978 INFO:tasks.workunit.client.0.smithi134.stdout: "num_scrub_errors": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_shallow_scrub_errors": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_deep_scrub_errors": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_recovered": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_recovered": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_keys_recovered": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_omap": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_hit_set_archive": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_bytes_hit_set_archive": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_kb": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_kb": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_promote": 0, 2024-04-25T09:14:50.979 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_high": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_flush_mode_low": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_some": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_evict_mode_full": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_pinned": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_legacy_snapsets": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_large_omap_objects": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_manifest": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_bytes": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_omap_keys": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "num_objects_repaired": 0 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "store_stats": { 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "total": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "available": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "internally_reserved": 0, 2024-04-25T09:14:50.980 INFO:tasks.workunit.client.0.smithi134.stdout: "allocated": 0, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "data_stored": 0, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed": 0, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_allocated": 0, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "data_compressed_original": 0, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "omap_allocated": 0, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "internal_metadata": 0 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "log_size": 0, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "ondisk_log_size": 0, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "up": 24, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "acting": 24, 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: "num_store_stats": 4 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.981 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: "osd_stats": [ 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 25769803856 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 55834574922 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T09:14:50.982 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 90194313279 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 3, 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: "seq": 124554051641 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_state": [ 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "active+recovery_unfound+degraded", 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 1 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.983 INFO:tasks.workunit.client.0.smithi134.stdout: "state": "active+clean", 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: "num": 8 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: "num_pg_by_osd": [ 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 3, 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 4, 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 7, 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 2, 2024-04-25T09:14:50.984 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 1, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 5, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 1, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 3, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 6, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "osd": 0, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "num_primary_pg": 1, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "num_acting_pg": 9, 2024-04-25T09:14:50.985 INFO:tasks.workunit.client.0.smithi134.stdout: "num_up_not_acting_pg": 0 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": [ 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 1, 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": {} 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: { 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: "pool": 2, 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: "purged_snaps": {} 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: ], 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: "servicemap": { 2024-04-25T09:14:50.986 INFO:tasks.workunit.client.0.smithi134.stdout: "epoch": 6, 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "modified": "2024-04-25T09:13:06.402836+0000", 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "services": {} 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: }, 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_first_committed": 1, 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "mgrstat_last_committed": 150, 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_first_committed": 1, 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "logm_last_committed": 640, 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "paxos": { 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "first_committed": 252, 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "last_committed": 831, 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "last_pn": 0, 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: "accepted_pn": 0 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout: } 2024-04-25T09:14:50.987 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T09:14:50.988 INFO:tasks.workunit.client.0.smithi134.stderr:report 2207574491 2024-04-25T09:14:50.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-04-25T09:14:50.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: ceph pg 2.0 query 2024-04-25T09:14:50.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:14:51.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:160: rados_get_data: COUNT=3 2024-04-25T09:14:51.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:161: rados_get_data: test 3 = 3 2024-04-25T09:14:51.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:162: rados_get_data: flush_pg_stats 2024-04-25T09:14:51.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:14:51.084 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:14:51.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:51.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:14:51.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803859 2024-04-25T09:14:51.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803859 2024-04-25T09:14:51.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803859' 2024-04-25T09:14:51.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:51.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:14:51.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574924 2024-04-25T09:14:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574924 2024-04-25T09:14:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803859 1-55834574924' 2024-04-25T09:14:51.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:51.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:14:51.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313282 2024-04-25T09:14:51.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313282 2024-04-25T09:14:51.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803859 1-55834574924 2-90194313282' 2024-04-25T09:14:51.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:14:51.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:14:52.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=124554051643 2024-04-25T09:14:52.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 124554051643 2024-04-25T09:14:52.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803859 1-55834574924 2-90194313282 3-124554051643' 2024-04-25T09:14:52.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:52.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803859 2024-04-25T09:14:52.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:52.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:14:52.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803859 2024-04-25T09:14:52.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:52.143 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803859 2024-04-25T09:14:52.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803859 2024-04-25T09:14:52.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803859' 2024-04-25T09:14:52.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:52.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803856 -lt 25769803859 2024-04-25T09:14:52.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:14:53.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:14:53.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:14:53.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803859 -lt 25769803859 2024-04-25T09:14:53.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:53.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574924 2024-04-25T09:14:53.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:53.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:14:53.978 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574924 2024-04-25T09:14:53.978 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:53.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574924 2024-04-25T09:14:53.979 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574924 2024-04-25T09:14:53.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574924' 2024-04-25T09:14:53.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:14:54.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574924 -lt 55834574924 2024-04-25T09:14:54.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:54.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:54.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-90194313282 2024-04-25T09:14:54.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:14:54.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-90194313282 2024-04-25T09:14:54.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:54.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313282 2024-04-25T09:14:54.436 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 90194313282 2024-04-25T09:14:54.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 90194313282' 2024-04-25T09:14:54.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:14:54.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313282 -lt 90194313282 2024-04-25T09:14:54.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:14:54.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-124554051643 2024-04-25T09:14:54.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:14:54.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:14:54.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-124554051643 2024-04-25T09:14:54.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:14:54.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=124554051643 2024-04-25T09:14:54.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 124554051643' 2024-04-25T09:14:54.848 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 124554051643 2024-04-25T09:14:54.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:14:55.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 124554051643 -lt 124554051643 2024-04-25T09:14:55.321 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: ceph pg dump --format=json-pretty 2024-04-25T09:14:55.321 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:14:55.721 INFO:tasks.workunit.client.0.smithi134.stderr:dumped all 2024-04-25T09:14:55.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:163: rados_get_data: COUNT=4 2024-04-25T09:14:55.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:164: rados_get_data: test 4 = 4 2024-04-25T09:14:55.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:177: TEST_rados_get_with_eio: delete_pool pool-rep 2024-04-25T09:14:55.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-04-25T09:14:55.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-04-25T09:14:56.414 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'pool-rep' does not exist 2024-04-25T09:14:56.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:14:56.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:14:56.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:14:56.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:14:56.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:14:56.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:14:56.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:14:56.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:14:56.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:14:56.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:14:56.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:14:56.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:14:56.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:14:56.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:14:56.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:14:56.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:14:56.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:14:56.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:14:56.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:14:56.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:14:56.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:14:56.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:56.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:14:56.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.390086 2024-04-25T09:14:56.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:14:56.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:14:56.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:14:56.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:14:56.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:14:56.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:14:56.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:14:56.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:14:56.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:14:56.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:14:56.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:14:56.791 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:14:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:14:56.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:14:56.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:14:56.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:14:56.794 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:14:56.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:14:56.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:14:56.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:14:56.795 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:14:56.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:14:56.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:14:56.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:14:56.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:56.799 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:14:56.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.390086 2024-04-25T09:14:56.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:14:56.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:14:56.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:14:56.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:56.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:14:56.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.390086 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:14:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:14:57.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:14:57.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:14:57.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:14:57.291 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:14:57.291 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:57.292 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:14:57.292 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:14:57.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:14:57.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:14:57.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:14:57.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:14:57.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:14:57.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:14:57.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:14:57.529 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:14:57.530 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:14:57.530 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:14:57.530 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:14:57.530 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:57.530 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:14:57.530 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-mon.a.asok 2024-04-25T09:14:57.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:14:57.531 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.390086/ceph-mon.a.asok config get fsid 2024-04-25T09:14:57.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:14:57.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:14:57.646 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:14:57.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-mon.a.asok 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:14:57.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.390086/ceph-mon.a.asok config get mon_host 2024-04-25T09:14:57.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:14:57.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:57.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:14:57.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:14:57.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:14:57.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:14:57.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:14:58.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:14:58.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:14:58.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:14:58.658 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:14:58.659 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:58.659 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:14:58.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:14:58.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:14:58.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:14:58.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:14:59.005 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T09:14:59.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rados_repair_warning td/osd-rep-recov-eio.sh 2024-04-25T09:14:59.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:181: TEST_rados_repair_warning: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:59.023 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: expr 10 + 1 2024-04-25T09:14:59.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:182: TEST_rados_repair_warning: local OBJS=11 2024-04-25T09:14:59.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:184: TEST_rados_repair_warning: setup_osds 4 2024-04-25T09:14:59.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=4 2024-04-25T09:14:59.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:14:59.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:14:59.025 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 4 - 1 2024-04-25T09:14:59.025 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 3 2024-04-25T09:14:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:14:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:14:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:14:59.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:14:59.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:14:59.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:14:59.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:14:59.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:14:59.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:14:59.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:14:59.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:14:59.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:14:59.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=02da0b50-4b37-4a1a-8d2a-c26c13e84d06 2024-04-25T09:14:59.032 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 02da0b50-4b37-4a1a-8d2a-c26c13e84d06 2024-04-25T09:14:59.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 02da0b50-4b37-4a1a-8d2a-c26c13e84d06' 2024-04-25T09:14:59.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:14:59.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATHypmJz7OAxAAweHUV2YaG+2W39ROUTZ7HA== 2024-04-25T09:14:59.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATHypmJz7OAxAAweHUV2YaG+2W39ROUTZ7HA=="}' 2024-04-25T09:14:59.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 02da0b50-4b37-4a1a-8d2a-c26c13e84d06 -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:14:59.398 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:14:59.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:14:59.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATHypmJz7OAxAAweHUV2YaG+2W39ROUTZ7HA== --osd-uuid 02da0b50-4b37-4a1a-8d2a-c26c13e84d06 2024-04-25T09:14:59.446 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:14:59.454+0000 7f3e1bc1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:59.475 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:14:59.486+0000 7f3e1bc1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:59.477 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:14:59.486+0000 7f3e1bc1a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:14:59.477 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:14:59.486+0000 7f3e1bc1a7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:14:59.477 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:14:59.486+0000 7f3e1bc1a7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:14:59.477 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:14:59.486+0000 7f3e1bc1a7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:14:59.515 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:14:59.526+0000 7f3e1bc1a7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:15:06.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:15:06.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:15:06.178 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:15:06.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:15:06.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:15:06.715 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:15:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:15:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:15:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:15:06.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:15:06.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:15:06.746 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:06.754+0000 7fc8fd68c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:06.748 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:06.758+0000 7fc8fd68c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:06.749 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:06.758+0000 7fc8fd68c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:07.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:15:07.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:15:07.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:15:07.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:15:07.118 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:15:07.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:15:07.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:07.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:15:07.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:07.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:07.300 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:07.310+0000 7fc8fd68c7c0 -1 Falling back to public interface 2024-04-25T09:15:07.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:08.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:08.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:08.572 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:15:08.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:15:08.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:08.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:08.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:09.963 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:09.974+0000 7fc8fd68c7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:15:09.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:09.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:09.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:15:09.976 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:15:09.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:09.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:10.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:11.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:11.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:11.638 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:15:11.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:15:11.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:11.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:12.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:13.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:13.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:13.066 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:15:13.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:15:13.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:13.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:13.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:14.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:14.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:14.704 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:15:14.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:15:14.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:14.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:15.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:16.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:16.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:16.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:15:16.110 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:15:16.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:16.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:16.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:17.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:17.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:17.569 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:15:17.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:15:17.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:17.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:18.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:19.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:19.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:19.108 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:15:19.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:15:19.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:19.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:19.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:20.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:20.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:20.543 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:15:20.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:15:20.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:20.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:21.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:22.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:22.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:22.094 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:15:22.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:15:22.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:22.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:22.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:23.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:23.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:23.524 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:15:23.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:15:23.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:23.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:24.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:25.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:25.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:25.450 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:15:25.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:15:25.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:25.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:26.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:27.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:27.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:27.424 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:15:27.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:15:27.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:27.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:27.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:28.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:28.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:28.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:15:28.976 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:15:28.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:28.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:29.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:30.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:30.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:30.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:15:30.640 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:15:30.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:30.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:31.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:32.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:32.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:32.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:15:32.823 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:15:32.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:32.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:33.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:34.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:34.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:34.892 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:15:34.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:15:34.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:34.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:35.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:36.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:36.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:36.423 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:15:36.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:15:36.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:36.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:15:36.838 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/58647625,v1:127.0.0.1:6803/58647625] [v2:127.0.0.1:6804/58647625,v1:127.0.0.1:6805/58647625] exists,up 02da0b50-4b37-4a1a-8d2a-c26c13e84d06 2024-04-25T09:15:36.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:15:36.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:15:36.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:15:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:15:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:15:36.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:15:36.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=86951d52-ab98-4442-8ab0-6c412631ae5c 2024-04-25T09:15:36.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 86951d52-ab98-4442-8ab0-6c412631ae5c' 2024-04-25T09:15:36.846 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 86951d52-ab98-4442-8ab0-6c412631ae5c 2024-04-25T09:15:36.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:15:36.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA4Hypmo4tcNBAAxntJ/SsxlUWWXZ+cj1P1OA== 2024-04-25T09:15:36.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA4Hypmo4tcNBAAxntJ/SsxlUWWXZ+cj1P1OA=="}' 2024-04-25T09:15:36.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 86951d52-ab98-4442-8ab0-6c412631ae5c -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:15:37.345 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:15:37.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:15:37.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA4Hypmo4tcNBAAxntJ/SsxlUWWXZ+cj1P1OA== --osd-uuid 86951d52-ab98-4442-8ab0-6c412631ae5c 2024-04-25T09:15:37.392 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:37.405+0000 7fcbaa86e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:37.394 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:37.405+0000 7fcbaa86e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:37.395 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:37.409+0000 7fcbaa86e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:37.396 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:37.409+0000 7fcbaa86e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:15:37.396 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:37.409+0000 7fcbaa86e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:15:37.396 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:37.409+0000 7fcbaa86e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:15:37.474 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:37.485+0000 7fcbaa86e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:15:44.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:15:44.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:15:44.230 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:15:44.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:15:44.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:15:44.793 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:15:44.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:15:44.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:15:44.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:15:44.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:15:44.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:15:44.822 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:44.833+0000 7fbd95cac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:44.858 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:44.869+0000 7fbd95cac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:44.860 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:44.873+0000 7fbd95cac7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:15:45.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:15:45.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:15:45.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:15:45.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:15:45.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:15:45.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:45.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:15:45.233 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:15:45.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:45.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:45.405 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:45.417+0000 7fbd95cac7c0 -1 Falling back to public interface 2024-04-25T09:15:45.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:46.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:46.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:46.648 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:15:46.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:15:46.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:46.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:47.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:48.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:48.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:48.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:15:48.055 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:15:48.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:48.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:48.145 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:15:48.157+0000 7fbd95cac7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:15:48.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:49.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:49.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:49.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:15:49.520 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:15:49.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:49.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:50.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:51.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:51.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:51.002 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:15:51.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:15:51.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:51.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:51.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:52.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:52.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:52.565 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:15:52.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:15:52.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:52.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:53.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:54.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:54.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:54.102 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:15:54.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:15:54.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:54.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:54.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:55.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:55.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:55.508 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:15:55.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:15:55.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:55.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:55.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:56.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:56.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:56.919 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:15:56.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:15:56.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:56.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:57.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:58.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:58.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:58.377 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:15:58.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:15:58.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:58.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:15:58.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:15:59.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:15:59.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:15:59.857 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:15:59.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:15:59.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:15:59.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:00.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:01.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:01.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:01.373 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:16:01.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:16:01.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:01.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:01.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:02.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:02.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:02.832 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:16:02.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:16:02.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:02.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:03.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:04.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:04.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:04.790 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:16:04.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:16:04.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:04.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:05.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:06.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:06.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:06.714 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:16:06.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:16:06.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:06.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:07.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:08.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:08.461 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:16:08.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:16:08.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:08.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:09.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:10.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:10.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:10.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:16:10.122 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:16:10.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:10.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:10.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:11.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:11.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:11.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:16:11.888 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:16:11.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:11.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:12.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:13.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:13.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:13.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:16:13.507 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:16:13.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:13.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:13.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:14.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:14.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:14.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:16:14.919 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T09:16:14.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:14.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:16:15.326 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3884478206,v1:127.0.0.1:6811/3884478206] [v2:127.0.0.1:6812/3884478206,v1:127.0.0.1:6813/3884478206] exists,up 86951d52-ab98-4442-8ab0-6c412631ae5c 2024-04-25T09:16:15.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:16:15.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:16:15.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:16:15.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:16:15.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:16:15.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:16:15.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:16:15.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:16:15.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:16:15.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:16:15.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:16:15.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:16:15.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:16:15.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:16:15.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:16:15.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:16:15.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:16:15.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:16:15.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:16:15.335 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:16:15.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0a33ce22-152f-41f0-a0ec-4714aa270832 2024-04-25T09:16:15.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 0a33ce22-152f-41f0-a0ec-4714aa270832' 2024-04-25T09:16:15.335 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 0a33ce22-152f-41f0-a0ec-4714aa270832 2024-04-25T09:16:15.336 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:16:15.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBfHypm/yiaFRAAKwq0TrPHU8zdbYUuaIyGXg== 2024-04-25T09:16:15.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBfHypm/yiaFRAAKwq0TrPHU8zdbYUuaIyGXg=="}' 2024-04-25T09:16:15.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0a33ce22-152f-41f0-a0ec-4714aa270832 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:16:15.857 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:16:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:16:15.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBfHypm/yiaFRAAKwq0TrPHU8zdbYUuaIyGXg== --osd-uuid 0a33ce22-152f-41f0-a0ec-4714aa270832 2024-04-25T09:16:15.903 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:15.916+0000 7fde399127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:15.905 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:15.916+0000 7fde399127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:15.906 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:15.920+0000 7fde399127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:15.907 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:15.920+0000 7fde399127c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:16:15.907 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:15.920+0000 7fde399127c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:16:15.907 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:15.920+0000 7fde399127c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:16:15.975 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:15.988+0000 7fde399127c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:16:23.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:16:23.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:16:23.045 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:16:23.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:16:23.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:16:23.613 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:16:23.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:16:23.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:16:23.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:16:23.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:16:23.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:16:23.643 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:23.656+0000 7f8688e947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:23.665 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:23.676+0000 7f8688e947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:23.666 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:23.680+0000 7f8688e947c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:24.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:16:24.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:16:24.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:16:24.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:16:24.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:16:24.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:24.062 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:16:24.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:16:24.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:24.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:24.228 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:24.240+0000 7f8688e947c0 -1 Falling back to public interface 2024-04-25T09:16:24.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:25.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:25.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:25.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:16:25.513 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:16:25.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:25.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:25.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:26.920 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:26.932+0000 7f8688e947c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:16:26.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:26.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:26.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:16:26.925 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:16:26.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:26.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:27.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:28.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:28.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:28.329 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:16:28.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:16:28.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:28.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:28.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:29.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:29.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:29.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:16:29.820 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:16:29.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:29.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:30.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:31.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:31.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:31.400 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:16:31.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:16:31.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:31.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:31.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:32.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:32.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:32.989 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:16:32.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:16:32.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:32.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:33.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:34.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:34.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:34.542 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:16:34.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:16:34.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:34.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:35.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:36.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:36.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:36.022 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:16:36.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:16:36.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:36.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:36.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:37.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:37.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:37.515 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:16:37.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:16:37.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:37.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:37.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:38.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:38.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:38.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:16:38.982 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:16:38.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:38.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:39.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:40.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:40.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:40.412 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:16:40.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:16:40.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:40.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:40.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:41.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:41.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:41.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:16:41.869 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:16:41.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:41.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:42.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:43.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:43.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:43.418 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:16:43.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:16:43.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:43.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:44.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:45.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:45.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:45.100 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:16:45.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:16:45.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:45.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:46.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:47.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:47.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:47.057 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:16:47.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:16:47.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:47.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:47.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:48.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:48.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:48.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:16:48.796 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:16:48.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:48.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:49.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:50.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:50.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:50.649 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:16:50.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:16:50.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:50.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:52.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:53.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:53.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:53.028 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:16:53.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:16:53.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:53.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:53.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:16:54.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:16:54.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:16:54.497 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T09:16:54.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:16:54.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:16:54.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:16:55.227 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 19 up_thru 19 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4034639660,v1:127.0.0.1:6819/4034639660] [v2:127.0.0.1:6820/4034639660,v1:127.0.0.1:6821/4034639660] exists,up 0a33ce22-152f-41f0-a0ec-4714aa270832 2024-04-25T09:16:55.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:16:55.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:16:55.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:16:55.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:16:55.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 3 2024-04-25T09:16:55.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:16:55.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/3 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 ' 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/3' 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/3/journal' 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:16:55.230 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:16:55.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:16:55.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:16:55.231 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:16:55.231 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:16:55.231 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:16:55.232 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:16:55.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:16:55.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:16:55.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:16:55.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:16:55.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/3 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:16:55.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=46f60425-abd1-45f1-9486-c5b0caae31d1 2024-04-25T09:16:55.234 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 46f60425-abd1-45f1-9486-c5b0caae31d1 2024-04-25T09:16:55.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 46f60425-abd1-45f1-9486-c5b0caae31d1' 2024-04-25T09:16:55.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:16:55.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCHHypmFcj6DxAAmMp44WDTZs4Ay9eAd5pwHw== 2024-04-25T09:16:55.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCHHypmFcj6DxAAmMp44WDTZs4Ay9eAd5pwHw=="}' 2024-04-25T09:16:55.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 46f60425-abd1-45f1-9486-c5b0caae31d1 -i td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:16:55.995 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:16:56.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/3/new.json 2024-04-25T09:16:56.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHHypmFcj6DxAAmMp44WDTZs4Ay9eAd5pwHw== --osd-uuid 46f60425-abd1-45f1-9486-c5b0caae31d1 2024-04-25T09:16:56.045 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:56.055+0000 7fb2b544e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:56.151 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:56.159+0000 7fb2b544e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:56.152 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:56.163+0000 7fb2b544e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:16:56.153 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:56.163+0000 7fb2b544e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:16:56.153 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:56.163+0000 7fb2b544e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:16:56.153 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:56.163+0000 7fb2b544e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/3/block: (2) No such file or directory 2024-04-25T09:16:56.242 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:16:56.251+0000 7fb2b544e7c0 -1 bluestore(td/osd-rep-recov-eio.sh/3) _read_fsid unparsable uuid 2024-04-25T09:17:02.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/3/keyring 2024-04-25T09:17:02.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:17:02.973 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T09:17:02.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:17:02.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:17:03.553 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T09:17:03.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:17:03.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/3 --osd-journal=td/osd-rep-recov-eio.sh/3/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:17:03.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:17:03.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:17:03.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:17:03.582 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:17:03.591+0000 7f4cb3fc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:03.600 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:17:03.611+0000 7f4cb3fc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:03.601 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:17:03.611+0000 7f4cb3fc67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:17:03.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:17:03.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:17:03.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:17:03.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:17:03.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:17:03.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:03.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:17:03.981 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:17:03.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:03.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:04.160 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:17:04.171+0000 7f4cb3fc67c0 -1 Falling back to public interface 2024-04-25T09:17:04.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:05.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:05.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:05.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:17:05.424 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:17:05.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:05.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:05.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:06.818 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:17:06.827+0000 7f4cb3fc67c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:17:06.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:06.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:06.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:17:06.834 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:17:06.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:06.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:07.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:08.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:08.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:08.708 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:17:08.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:17:08.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:08.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:09.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:10.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:10.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:10.712 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:17:10.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:17:10.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:10.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:11.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:12.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:12.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:12.156 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:17:12.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:17:12.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:12.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:12.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:13.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:13.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:13.709 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:17:13.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:17:13.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:13.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:14.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:15.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:15.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:15.249 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:17:15.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:17:15.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:15.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:15.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:16.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:16.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:16.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:17:16.794 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:17:16.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:16.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:17.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:18.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:18.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:18.239 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:17:18.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:17:18.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:18.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:18.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:19.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:19.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:19.759 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:17:19.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:17:19.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:19.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:20.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:21.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:21.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:21.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:17:21.216 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:17:21.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:21.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:21.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:22.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:22.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:22.736 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:17:22.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:17:22.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:22.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:23.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:24.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:24.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:24.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:17:24.616 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:17:24.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:24.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:25.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:26.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:26.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:26.484 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:17:26.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:17:26.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:26.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:27.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:28.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:28.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:28.379 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:17:28.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:17:28.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:28.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:29.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:30.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:30.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:30.220 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:17:30.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:17:30.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:30.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:31.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:32.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:32.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:32.206 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:17:32.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:17:32.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:32.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:33.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:17:34.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:17:34.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:17:34.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:17:34.367 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:17:34.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:17:34.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:17:34.787 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 28 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/4174172663,v1:127.0.0.1:6827/4174172663] [v2:127.0.0.1:6828/4174172663,v1:127.0.0.1:6829/4174172663] exists,up 46f60425-abd1-45f1-9486-c5b0caae31d1 2024-04-25T09:17:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:17:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:17:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:17:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:17:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:17:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:17:34.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:17:34.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:17:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:17:34.789 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:17:34.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:17:34.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:17:34.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:17:34.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:17:34.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:17:34.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:17:34.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:17:34.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:17:34.889 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:17:35.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:17:35.322 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:35.322 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:17:35.322 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:17:35.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:17:35.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:35.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:17:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803807 2024-04-25T09:17:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803807 2024-04-25T09:17:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803807' 2024-04-25T09:17:35.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:35.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:17:35.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607574 2024-04-25T09:17:35.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607574 2024-04-25T09:17:35.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803807 1-51539607574' 2024-04-25T09:17:35.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:35.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:17:36.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378640 2024-04-25T09:17:36.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378640 2024-04-25T09:17:36.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803807 1-51539607574 2-81604378640' 2024-04-25T09:17:36.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:36.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:17:36.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=120259084299 2024-04-25T09:17:36.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 120259084299 2024-04-25T09:17:36.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803807 1-51539607574 2-81604378640 3-120259084299' 2024-04-25T09:17:36.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:36.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803807 2024-04-25T09:17:36.193 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:36.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:17:36.195 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803807 2024-04-25T09:17:36.195 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:36.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803807 2024-04-25T09:17:36.196 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803807 2024-04-25T09:17:36.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803807' 2024-04-25T09:17:36.197 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:17:36.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803805 -lt 25769803807 2024-04-25T09:17:36.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:17:37.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:17:37.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:17:38.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803808 -lt 25769803807 2024-04-25T09:17:38.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:38.366 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607574 2024-04-25T09:17:38.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:38.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:17:38.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:38.369 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607574 2024-04-25T09:17:38.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607574 2024-04-25T09:17:38.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607574' 2024-04-25T09:17:38.370 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 51539607574 2024-04-25T09:17:38.371 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:17:39.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607574 -lt 51539607574 2024-04-25T09:17:39.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:39.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-81604378640 2024-04-25T09:17:39.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:39.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:17:39.021 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-81604378640 2024-04-25T09:17:39.021 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:39.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378640 2024-04-25T09:17:39.022 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 81604378640 2024-04-25T09:17:39.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 81604378640' 2024-04-25T09:17:39.022 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:17:39.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378640 -lt 81604378640 2024-04-25T09:17:39.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:39.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-120259084299 2024-04-25T09:17:39.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:39.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:17:39.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-120259084299 2024-04-25T09:17:39.490 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:39.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=120259084299 2024-04-25T09:17:39.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 120259084299' 2024-04-25T09:17:39.492 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 120259084299 2024-04-25T09:17:39.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:17:39.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 120259084299 -lt 120259084299 2024-04-25T09:17:39.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:17:39.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:39.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:40.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:17:40.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:17:40.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:17:40.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:17:40.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:17:40.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:17:40.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:17:40.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:17:40.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:17:40.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:17:40.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:40.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:41.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 8 2024-04-25T09:17:41.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' -1 2024-04-25T09:17:41.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:17:41.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-04-25T09:17:41.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:17:41.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:17:41.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:17:41.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:17:41.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:17:41.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:17:41.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:17:41.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:17:41.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:17:41.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:17:41.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-04-25T09:17:41.908 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:17:41.908 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:41.908 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:42.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-04-25T09:17:42.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' 2 2024-04-25T09:17:42.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:17:42.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=7 2024-04-25T09:17:42.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:17:42.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:17:42.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:17:42.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:17:42.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:17:42.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:17:42.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:17:42.550 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:17:42.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:17:42.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:17:42.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=7 2024-04-25T09:17:42.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:17:42.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:42.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:43.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 7 = 8 2024-04-25T09:17:43.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 7 '!=' 7 2024-04-25T09:17:43.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:17:43.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:17:43.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:17:43.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:17:43.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:17:43.511 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:17:43.511 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:17:44.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:17:44.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:17:44.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T09:17:44.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:17:44.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T09:17:44.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:17:44.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:17:44.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:17:44.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:17:44.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:17:44.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:17:44.246 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:17:44.246 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:17:44.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:17:44.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:17:44.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:44.633 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:45.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:17:45.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:17:45.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:17:45.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:186: TEST_rados_repair_warning: local poolname=pool-rep 2024-04-25T09:17:45.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:187: TEST_rados_repair_warning: create_pool pool-rep 1 1 2024-04-25T09:17:45.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create pool-rep 1 1 2024-04-25T09:17:45.685 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'pool-rep' already exists 2024-04-25T09:17:45.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:17:46.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:188: TEST_rados_repair_warning: wait_for_clean 2024-04-25T09:17:46.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:17:46.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:17:46.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:17:46.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:17:46.704 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:17:46.705 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:17:46.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:17:46.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:17:46.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:17:46.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:17:46.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:17:46.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:17:46.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:17:46.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:17:46.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:17:47.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:17:47.340 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:47.340 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:17:47.340 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:17:47.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:17:47.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:47.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:17:47.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803812 2024-04-25T09:17:47.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803812 2024-04-25T09:17:47.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803812' 2024-04-25T09:17:47.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:47.494 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:17:47.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607578 2024-04-25T09:17:47.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607578 2024-04-25T09:17:47.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803812 1-51539607578' 2024-04-25T09:17:47.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:47.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:17:47.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378644 2024-04-25T09:17:47.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378644 2024-04-25T09:17:47.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803812 1-51539607578 2-81604378644' 2024-04-25T09:17:47.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:17:47.844 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:17:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=120259084303 2024-04-25T09:17:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 120259084303 2024-04-25T09:17:47.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803812 1-51539607578 2-81604378644 3-120259084303' 2024-04-25T09:17:47.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:47.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803812 2024-04-25T09:17:47.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:47.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:17:47.994 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803812 2024-04-25T09:17:47.994 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:47.995 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803812 2024-04-25T09:17:47.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803812 2024-04-25T09:17:47.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803812' 2024-04-25T09:17:47.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:17:48.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803810 -lt 25769803812 2024-04-25T09:17:48.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:17:49.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:17:49.394 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:17:49.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803812 -lt 25769803812 2024-04-25T09:17:49.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:49.832 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607578 2024-04-25T09:17:49.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:49.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:17:49.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607578 2024-04-25T09:17:49.835 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:49.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607578 2024-04-25T09:17:49.836 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 51539607578 2024-04-25T09:17:49.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607578' 2024-04-25T09:17:49.837 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:17:50.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607578 -lt 51539607578 2024-04-25T09:17:50.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:50.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:50.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-81604378644 2024-04-25T09:17:50.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:17:50.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-81604378644 2024-04-25T09:17:50.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:50.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378644 2024-04-25T09:17:50.291 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 81604378644 2024-04-25T09:17:50.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 81604378644' 2024-04-25T09:17:50.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:17:50.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378644 -lt 81604378644 2024-04-25T09:17:50.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:17:50.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-120259084303 2024-04-25T09:17:50.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:17:50.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:17:50.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-120259084303 2024-04-25T09:17:50.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:17:50.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=120259084303 2024-04-25T09:17:50.701 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 120259084303 2024-04-25T09:17:50.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 120259084303' 2024-04-25T09:17:50.702 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:17:51.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 120259084303 -lt 120259084303 2024-04-25T09:17:51.126 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:17:51.126 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:51.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:51.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:17:51.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:17:51.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:17:51.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:17:51.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:17:51.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:17:51.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:17:51.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:17:52.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:17:52.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:17:52.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:17:52.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:17:52.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:17:52.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:17:52.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:17:52.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:190: TEST_rados_repair_warning: local poolname=pool-rep 2024-04-25T09:17:52.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:191: TEST_rados_repair_warning: local objbase=obj-warn 2024-04-25T09:17:52.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:192: TEST_rados_repair_warning: local inject=eio 2024-04-25T09:17:52.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: seq 1 11 2024-04-25T09:17:52.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-1 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:52.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:52.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:52.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:52.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:52.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:52.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:52.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T09:17:52.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:17:52.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:17:52.797 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:17:52.797 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:53.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:17:53.241 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:53.241 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:17:53.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:17:53.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:17:53.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:53.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:17:53.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:53.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:53.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:17:53.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:53.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:17:53.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:53.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:17:53.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:53.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:53.243 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:53.243 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:17:53.243 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:53.244 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:53.244 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:53.244 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:53.244 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:53.244 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:53.244 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:53.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:53.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:53.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:17:53.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:53.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:53.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:53.365 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:53.365 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:53.365 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:53.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:53.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:53.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-04-25T09:17:53.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T09:17:53.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:53.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:17:53.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-04-25T09:17:53.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:17:53.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:17:53.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:17:53.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:53.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:53.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-2 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:53.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:53.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:53.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:17:53.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T09:17:53.725 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T09:17:53.725 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:54.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:17:54.131 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:54.131 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:17:54.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:17:54.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:17:54.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:54.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:17:54.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:54.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:54.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:17:54.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:54.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:17:54.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:54.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:17:54.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:54.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:54.133 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:17:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:54.135 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:54.135 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:54.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:54.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:54.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:54.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:17:54.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:54.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:54.257 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:54.257 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:54.257 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:54.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:54.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:54.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-04-25T09:17:54.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T09:17:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:17:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-04-25T09:17:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:17:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:17:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:17:54.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:54.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:54.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-3 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:54.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:54.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:54.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:54.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:54.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:54.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:17:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T09:17:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:54.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:54.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:54.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:54.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:54.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T09:17:54.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:17:54.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T09:17:54.669 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T09:17:54.670 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:55.039 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:17:55.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:55.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:17:55.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:55.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:17:55.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:55.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:55.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:55.042 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:17:55.042 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:55.042 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:55.042 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:55.042 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:55.042 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:55.042 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:55.042 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:55.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:55.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:55.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:17:55.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:55.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:55.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:55.159 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:55.159 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:55.159 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:55.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:55.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:55.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-04-25T09:17:55.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T09:17:55.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:55.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:17:55.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-04-25T09:17:55.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:17:55.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:17:55.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:17:55.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:55.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:55.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:55.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:17:55.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T09:17:55.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-4 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:55.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:55.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:55.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:55.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:55.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:55.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T09:17:55.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:17:55.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T09:17:55.526 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T09:17:55.526 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:55.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:17:55.947 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:55.947 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:17:55.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:17:55.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:17:55.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:55.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:17:55.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:55.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:55.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:17:55.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:55.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:17:55.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:55.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:17:55.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:55.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:55.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:55.952 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:17:55.953 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:55.953 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:55.953 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:55.953 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:55.953 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:55.953 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:55.953 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:56.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:56.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:56.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:56.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:17:56.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:56.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:56.073 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:56.073 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:56.074 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:56.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:56.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:56.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-04-25T09:17:56.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T09:17:56.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:56.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:17:56.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-04-25T09:17:56.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:17:56.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:17:56.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:17:56.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:56.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:56.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-5 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:56.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:56.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:56.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:17:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T09:17:56.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:56.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:56.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:56.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:56.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:56.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T09:17:56.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:17:56.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T09:17:56.425 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T09:17:56.426 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:56.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:17:56.833 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:56.833 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:17:56.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:17:56.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:17:56.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:56.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:17:56.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:56.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:56.834 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:17:56.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:56.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:17:56.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:56.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:17:56.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:56.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:56.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:17:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:56.836 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:56.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:56.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:56.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:56.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:17:56.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:56.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:56.960 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:56.960 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:56.960 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:56.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:56.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:56.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-04-25T09:17:57.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T09:17:57.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:57.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:17:57.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-04-25T09:17:57.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:17:57.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:17:57.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:17:57.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:57.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:57.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-6 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:57.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:57.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:57.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:57.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:57.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T09:17:57.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:17:57.287 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T09:17:57.287 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:57.287 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T09:17:57.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:17:57.709 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:57.709 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:17:57.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:17:57.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:17:57.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:57.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:17:57.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:57.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:57.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:17:57.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:57.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:17:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:17:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:57.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:57.713 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:17:57.714 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:57.714 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:57.714 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:57.714 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:57.714 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:57.714 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:57.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:57.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:57.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:57.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:57.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:17:57.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:57.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:57.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:57.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:57.831 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:57.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:57.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:57.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-04-25T09:17:57.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T09:17:57.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:57.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:17:57.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-04-25T09:17:57.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:17:57.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:17:57.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:17:57.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:58.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:58.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:58.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:17:58.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T09:17:58.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:58.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:17:58.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-7 2024-04-25T09:17:58.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:58.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:58.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:58.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:58.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:58.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:58.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:58.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:58.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:58.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:58.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:58.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:58.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:58.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:17:58.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T09:17:58.207 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T09:17:58.207 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:58.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:17:58.618 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:58.618 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:17:58.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:17:58.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:17:58.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:58.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:17:58.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:58.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:58.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:17:58.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:58.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:17:58.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:58.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:17:58.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:58.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:58.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:58.621 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:17:58.622 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:58.622 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:58.622 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:58.622 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:58.622 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:58.622 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:58.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:58.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:58.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:58.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:58.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:17:58.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:58.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:58.733 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:58.733 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:58.734 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:58.734 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:58.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:58.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-04-25T09:17:58.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T09:17:58.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:58.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:17:58.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-04-25T09:17:58.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:17:58.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:17:58.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:17:58.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:58.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:58.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:58.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:17:58.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T09:17:58.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:58.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:17:58.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-8 2024-04-25T09:17:58.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:58.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:58.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:58.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:58.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:58.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:58.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:58.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:58.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:59.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T09:17:59.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:17:59.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T09:17:59.066 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T09:17:59.066 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:17:59.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:17:59.492 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:17:59.492 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:17:59.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:17:59.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:17:59.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:17:59.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:17:59.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:17:59.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:17:59.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:17:59.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:17:59.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:17:59.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:17:59.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:17:59.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:17:59.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:17:59.494 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:17:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:17:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:59.496 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:17:59.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:17:59.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:17:59.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:17:59.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:17:59.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:17:59.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:17:59.619 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:17:59.619 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:17:59.619 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:17:59.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:17:59.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:17:59.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-04-25T09:17:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T09:17:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:17:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-04-25T09:17:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:17:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:17:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:17:59.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:59.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:59.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-9 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:17:59.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:17:59.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:17:59.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:17:59.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-04-25T09:17:59.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:17:59.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:17:59.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:17:59.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:17:59.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:17:59.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:17:59.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T09:17:59.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:17:59.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:17:59.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:17:59.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:17:59.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:17:59.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T09:17:59.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:17:59.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T09:17:59.997 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T09:17:59.997 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:00.456 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:00.456 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:00.456 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:00.457 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:00.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:00.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:00.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:00.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:00.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:00.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:00.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:00.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:00.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:00.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:00.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:00.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:00.459 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:00.459 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:00.459 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:00.459 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:00.460 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:00.460 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:00.460 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:00.460 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:00.460 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:00.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:00.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:00.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:00.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:00.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:00.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:00.585 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:00.585 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:00.585 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:00.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:00.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:00.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-04-25T09:18:00.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T09:18:00.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:00.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:00.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-04-25T09:18:00.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:00.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:00.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:00.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:00.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:00.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-10 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:18:00.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:18:00.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:18:00.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:18:00.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:18:01.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:01.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:01.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T09:18:01.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:01.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:01.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:01.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:01.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:01.018 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T09:18:01.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:01.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T09:18:01.019 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T09:18:01.019 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:01.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:01.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:01.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:01.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:01.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:01.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:01.474 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:01.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:01.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:01.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:01.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:01.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:01.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:01.591 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:01.591 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:01.591 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:01.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:01.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:01.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-04-25T09:18:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T09:18:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-04-25T09:18:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:01.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:01.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:01.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:194: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:01.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:196: TEST_rados_repair_warning: rados_put td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T09:18:01.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=pool-rep 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=obj-warn-11 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:18:01.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool pool-rep put obj-warn-11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:18:01.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:197: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:01.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:01.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T09:18:01.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:01.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T09:18:01.923 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T09:18:01.923 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:02.396 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:02.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:02.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:02.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:02.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:02.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:02.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:02.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:02.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:02.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:02.399 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:02.399 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:02.399 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:02.400 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:02.400 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:02.400 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:02.400 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:02.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:02.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:02.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:02.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:02.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:02.522 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:02.522 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:02.523 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:02.523 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:02.523 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:02.523 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:02.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:02.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-04-25T09:18:02.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:198: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T09:18:02.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:02.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:02.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-04-25T09:18:02.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:02.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:02.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:02.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:02.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:02.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:02.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: get_pg pool-rep obj-warn-1 2024-04-25T09:18:02.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=pool-rep 2024-04-25T09:18:02.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=obj-warn-1 2024-04-25T09:18:02.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:18:02.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-04-25T09:18:03.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:200: TEST_rados_repair_warning: local pgid=2.0 2024-04-25T09:18:03.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: get_osds pool-rep obj-warn-1 2024-04-25T09:18:03.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:03.103 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:18:03.104 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:18:03.104 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:03.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:03.500 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:03.500 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:03.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:03.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: object_osds=('3' '1' '0') 2024-04-25T09:18:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:202: TEST_rados_repair_warning: local object_osds 2024-04-25T09:18:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:203: TEST_rados_repair_warning: local primary=3 2024-04-25T09:18:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:204: TEST_rados_repair_warning: local bad_peer=1 2024-04-25T09:18:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:206: TEST_rados_repair_warning: wait_for_clean 2024-04-25T09:18:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:18:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:18:03.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:18:03.501 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:18:03.502 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:18:03.502 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:18:03.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:18:03.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:18:03.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:18:03.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:18:03.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:18:03.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:18:03.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:18:03.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:18:03.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:18:04.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:18:04.011 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:04.011 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:18:04.011 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:18:04.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:18:04.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:04.011 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:18:04.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803817 2024-04-25T09:18:04.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803817 2024-04-25T09:18:04.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817' 2024-04-25T09:18:04.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:04.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:18:04.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607583 2024-04-25T09:18:04.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607583 2024-04-25T09:18:04.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-51539607583' 2024-04-25T09:18:04.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:04.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:18:04.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378649 2024-04-25T09:18:04.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378649 2024-04-25T09:18:04.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-51539607583 2-81604378649' 2024-04-25T09:18:04.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:04.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:18:04.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=120259084309 2024-04-25T09:18:04.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 120259084309 2024-04-25T09:18:04.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803817 1-51539607583 2-81604378649 3-120259084309' 2024-04-25T09:18:04.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:04.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803817 2024-04-25T09:18:04.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:04.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:18:04.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803817 2024-04-25T09:18:04.619 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:04.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803817 2024-04-25T09:18:04.620 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803817 2024-04-25T09:18:04.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803817' 2024-04-25T09:18:04.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:05.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803815 -lt 25769803817 2024-04-25T09:18:05.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:18:06.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:18:06.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803817 -lt 25769803817 2024-04-25T09:18:06.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:06.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607583 2024-04-25T09:18:06.432 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:06.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:18:06.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:06.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607583 2024-04-25T09:18:06.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607583 2024-04-25T09:18:06.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607583' 2024-04-25T09:18:06.435 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 51539607583 2024-04-25T09:18:06.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:18:06.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607584 -lt 51539607583 2024-04-25T09:18:06.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:06.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-81604378649 2024-04-25T09:18:06.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:06.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:18:06.872 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-81604378649 2024-04-25T09:18:06.872 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:06.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378649 2024-04-25T09:18:06.875 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 81604378649 2024-04-25T09:18:06.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 81604378649' 2024-04-25T09:18:06.875 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:18:07.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378650 -lt 81604378649 2024-04-25T09:18:07.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:07.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-120259084309 2024-04-25T09:18:07.298 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:07.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:18:07.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-120259084309 2024-04-25T09:18:07.300 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:07.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=120259084309 2024-04-25T09:18:07.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 120259084309' 2024-04-25T09:18:07.302 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 120259084309 2024-04-25T09:18:07.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:18:07.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 120259084309 -lt 120259084309 2024-04-25T09:18:07.699 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:18:07.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:18:07.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:18:08.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:18:08.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:18:08.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:18:08.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:18:08.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:18:08.238 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:18:08.238 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:18:08.238 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:18:08.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:18:08.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:18:08.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:18:08.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:18:09.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:18:09.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:18:09.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:18:09.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: ceph pg 2.0 query 2024-04-25T09:18:09.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:18:09.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:207: TEST_rados_repair_warning: COUNT=11 2024-04-25T09:18:09.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:208: TEST_rados_repair_warning: test 11 = 11 2024-04-25T09:18:09.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:209: TEST_rados_repair_warning: flush_pg_stats 2024-04-25T09:18:09.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:18:09.328 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:18:09.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:18:09.725 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:09.725 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:18:09.725 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:18:09.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:18:09.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:09.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:18:09.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803820 2024-04-25T09:18:09.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803820 2024-04-25T09:18:09.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820' 2024-04-25T09:18:09.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:09.872 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:18:10.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607587 2024-04-25T09:18:10.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607587 2024-04-25T09:18:10.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-51539607587' 2024-04-25T09:18:10.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:10.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:18:10.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378653 2024-04-25T09:18:10.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378653 2024-04-25T09:18:10.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-51539607587 2-81604378653' 2024-04-25T09:18:10.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:10.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:18:10.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=120259084312 2024-04-25T09:18:10.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 120259084312 2024-04-25T09:18:10.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803820 1-51539607587 2-81604378653 3-120259084312' 2024-04-25T09:18:10.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:10.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803820 2024-04-25T09:18:10.282 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:10.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:18:10.283 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803820 2024-04-25T09:18:10.284 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:10.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803820 2024-04-25T09:18:10.285 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803820 2024-04-25T09:18:10.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803820' 2024-04-25T09:18:10.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:10.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803818 -lt 25769803820 2024-04-25T09:18:10.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:18:11.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:18:11.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:12.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803820 -lt 25769803820 2024-04-25T09:18:12.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:12.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607587 2024-04-25T09:18:12.129 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:12.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:18:12.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607587 2024-04-25T09:18:12.130 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:12.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607587 2024-04-25T09:18:12.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607587' 2024-04-25T09:18:12.131 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 51539607587 2024-04-25T09:18:12.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:18:12.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607587 -lt 51539607587 2024-04-25T09:18:12.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:12.558 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-81604378653 2024-04-25T09:18:12.558 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:12.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:18:12.560 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-81604378653 2024-04-25T09:18:12.560 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:12.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378653 2024-04-25T09:18:12.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 81604378653' 2024-04-25T09:18:12.562 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 81604378653 2024-04-25T09:18:12.562 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:18:13.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378653 -lt 81604378653 2024-04-25T09:18:13.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:13.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-120259084312 2024-04-25T09:18:13.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:13.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:18:13.058 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:13.059 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-120259084312 2024-04-25T09:18:13.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=120259084312 2024-04-25T09:18:13.059 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 120259084312 2024-04-25T09:18:13.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 120259084312' 2024-04-25T09:18:13.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:18:13.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 120259084312 -lt 120259084312 2024-04-25T09:18:13.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-04-25T09:18:13.518 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:18:13.916 INFO:tasks.workunit.client.0.smithi134.stderr:dumped all 2024-04-25T09:18:13.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:210: TEST_rados_repair_warning: COUNT=11 2024-04-25T09:18:13.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:211: TEST_rados_repair_warning: test 11 = 11 2024-04-25T09:18:13.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: ceph health 2024-04-25T09:18:13.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:213: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-04-25T09:18:14.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: grep -q 'osd.3 had 11 reads repaired' 2024-04-25T09:18:14.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:214: TEST_rados_repair_warning: ceph health detail 2024-04-25T09:18:15.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:216: TEST_rados_repair_warning: ceph health mute OSD_TOO_MANY_REPAIRS 2024-04-25T09:18:15.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:217: TEST_rados_repair_warning: set -o pipefail 2024-04-25T09:18:15.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: ceph health 2024-04-25T09:18:15.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:219: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 1 OSDs' 2024-04-25T09:18:16.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:220: TEST_rados_repair_warning: set +o pipefail 2024-04-25T09:18:16.186 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: seq 1 11 2024-04-25T09:18:16.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:16.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:16.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:16.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:16.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:16.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:16.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:16.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:16.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T09:18:16.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:16.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:16.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:16.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:16.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:16.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T09:18:16.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:16.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:18:16.189 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:16.189 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:18:16.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:16.607 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:16.607 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:16.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:16.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:16.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:16.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:16.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:16.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:16.607 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:16.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:16.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:16.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:16.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:16.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:16.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:16.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:16.610 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:16.610 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:16.610 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:16.610 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:16.610 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:16.610 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:16.610 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:16.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:16.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:16.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:16.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:16.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:16.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:16.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:16.733 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:16.733 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:16.733 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:16.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:16.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:16.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-1 2024-04-25T09:18:16.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-1 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:16.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:16.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:16.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:16.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:16.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:16.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:16.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-1 2024-04-25T09:18:16.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:16.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:16.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:16.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:16.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:16.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-1 2024-04-25T09:18:16.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:16.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-1 2024-04-25T09:18:16.853 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-1 2024-04-25T09:18:16.854 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:17.269 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:17.269 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:17.269 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:17.270 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:17.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:17.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:17.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:17.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:17.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:17.270 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:17.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:17.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:17.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:17.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:17.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:17.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:17.274 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:17.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:17.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:17.274 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:17.274 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:17.275 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:17.275 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:17.275 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:17.275 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:17.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:17.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:17.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:17.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:17.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:17.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:17.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:17.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:17.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:17.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:17.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:17.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-1 2024-04-25T09:18:17.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:17.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:17.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:17.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:17.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:17.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:17.508 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:17.508 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:17.508 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:17.509 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:17.509 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:17.509 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:17.509 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:17.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:17.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:17.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-1 2024-04-25T09:18:17.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:17.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:17.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-1 2024-04-25T09:18:17.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:17.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:17.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:17.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-1 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:17.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:17.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:17.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T09:18:17.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:17.742 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T09:18:17.742 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T09:18:17.742 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:18.162 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:18.162 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:18.162 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:18.162 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:18.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:18.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:18.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:18.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:18.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:18.163 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:18.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:18.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:18.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:18.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:18.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:18.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:18.164 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:18.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:18.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:18.165 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:18.165 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:18.165 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:18.166 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:18.166 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:18.167 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:18.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:18.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:18.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:18.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:18.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:18.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:18.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:18.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:18.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:18.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:18.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:18.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-2 2024-04-25T09:18:18.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-2 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:18.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:18.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:18.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:18.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:18.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:18.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:18.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-2 2024-04-25T09:18:18.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:18.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:18.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:18.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:18.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:18.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-2 2024-04-25T09:18:18.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:18.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-2 2024-04-25T09:18:18.410 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-2 2024-04-25T09:18:18.410 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:18.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:18.829 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:18.829 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:18.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:18.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:18.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:18.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:18.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:18.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:18.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:18.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:18.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:18.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:18.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:18.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:18.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:18.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:18.832 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:18.832 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:18.832 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:18.833 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:18.833 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:18.833 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:18.833 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:18.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:18.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:18.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:18.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:18.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:18.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:18.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:18.956 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:18.956 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:18.956 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:18.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:18.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:18.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-2 2024-04-25T09:18:19.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:19.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:19.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:19.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:19.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:19.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:19.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:19.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:19.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:19.068 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:19.068 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:19.068 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:19.068 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:19.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:19.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:19.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-2 2024-04-25T09:18:19.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:19.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:19.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-2 2024-04-25T09:18:19.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:19.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:19.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:19.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-2 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:19.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:19.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:19.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T09:18:19.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:19.265 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T09:18:19.265 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:19.265 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T09:18:19.668 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:19.669 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:19.669 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:19.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:19.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:19.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:19.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:19.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:19.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:19.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:19.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:19.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:19.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:19.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:19.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:19.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:19.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:19.672 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:19.672 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:19.672 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:19.672 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:19.673 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:19.673 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:19.673 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:19.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:19.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:19.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:19.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:19.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:19.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:19.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:19.788 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:19.788 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:19.789 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:19.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:19.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:19.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-3 2024-04-25T09:18:19.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-3 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:19.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:19.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:19.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:19.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:19.901 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:19.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:19.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-3 2024-04-25T09:18:19.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:19.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:19.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:19.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:19.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:19.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-3 2024-04-25T09:18:19.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:19.903 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-3 2024-04-25T09:18:19.903 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:19.903 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-3 2024-04-25T09:18:20.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:20.374 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:20.375 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:20.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:20.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:20.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:20.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:20.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:20.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:20.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:20.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:20.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:20.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:20.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:20.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:20.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:20.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:20.377 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:20.377 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:20.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:20.378 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:20.378 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:20.378 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:20.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:20.378 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:20.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:20.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:20.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:20.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:20.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:20.504 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:20.504 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:20.504 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:20.505 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:20.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:20.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:20.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-3 2024-04-25T09:18:20.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:20.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:20.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:20.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:20.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:20.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:20.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:20.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:20.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:20.629 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:20.629 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:20.629 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:20.629 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:20.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:20.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:20.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-3 2024-04-25T09:18:20.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:20.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:20.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-3 2024-04-25T09:18:20.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:20.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:20.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:20.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-3 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:20.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:20.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:20.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:20.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:20.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:20.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T09:18:20.807 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T09:18:20.808 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:21.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:21.286 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:21.286 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:21.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:21.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:21.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:21.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:21.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:21.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:21.287 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:21.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:21.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:21.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:21.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:21.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:21.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:21.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:21.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:21.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:21.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:21.289 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:21.289 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:21.290 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:21.290 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:21.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:21.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:21.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:21.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:21.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:21.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:21.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:21.409 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:21.409 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:21.409 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:21.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:21.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:21.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-4 2024-04-25T09:18:21.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-4 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:21.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:21.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-4 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:21.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-4 2024-04-25T09:18:21.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:21.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-4 2024-04-25T09:18:21.526 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-4 2024-04-25T09:18:21.526 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:21.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:21.949 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:21.949 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:21.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:21.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:21.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:21.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:21.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:21.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:21.950 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:21.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:21.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:21.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:21.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:21.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:21.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:21.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:21.955 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:21.955 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:21.955 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:21.955 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:21.955 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:21.956 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:21.956 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:21.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:22.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:22.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:22.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:22.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:22.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:22.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:22.069 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:22.069 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:22.069 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:22.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:22.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:22.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-4 2024-04-25T09:18:22.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:22.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:22.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:22.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:22.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:22.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:22.178 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:22.178 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:22.178 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:22.178 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:22.178 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:22.178 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:22.179 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:22.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:22.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:22.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-4 2024-04-25T09:18:22.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:22.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:22.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-4 2024-04-25T09:18:22.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:22.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:22.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:22.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-4 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:22.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:22.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:22.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:22.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:22.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T09:18:22.348 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T09:18:22.349 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:22.744 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:22.745 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:22.745 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:22.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:22.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:22.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:22.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:22.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:22.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:22.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:22.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:22.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:22.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:22.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:22.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:22.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:22.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:22.747 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:22.747 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:22.747 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:22.748 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:22.748 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:22.748 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:22.748 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:22.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:22.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:22.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:22.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:22.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:22.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:22.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:22.871 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:22.871 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:22.871 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:22.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:22.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:22.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-5 2024-04-25T09:18:22.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-5 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:22.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:22.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:22.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:22.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:22.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:22.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:22.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-5 2024-04-25T09:18:22.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:22.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:22.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:22.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:22.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:22.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-5 2024-04-25T09:18:22.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:22.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-5 2024-04-25T09:18:22.994 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-5 2024-04-25T09:18:22.994 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:23.438 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:23.439 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:23.439 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:23.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:23.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:23.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:23.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:23.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:23.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:23.439 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:23.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:23.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:23.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:23.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:23.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:23.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:23.441 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:23.441 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:23.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:23.441 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:23.442 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:23.442 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:23.442 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:23.443 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:23.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:23.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:23.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:23.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:23.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:23.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:23.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:23.564 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:23.564 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:23.564 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:23.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:23.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:23.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-5 2024-04-25T09:18:23.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:23.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:23.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:23.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:23.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:23.685 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:23.686 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:23.686 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:23.686 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:23.687 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:23.687 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:23.687 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:23.687 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:23.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:23.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:23.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-5 2024-04-25T09:18:23.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:23.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:23.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-5 2024-04-25T09:18:23.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:23.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:23.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:23.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-5 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:23.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:23.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:23.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:23.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:23.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:23.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T09:18:23.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:23.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T09:18:23.879 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T09:18:23.879 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:24.285 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:24.285 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:24.285 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:24.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:24.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:24.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:24.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:24.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:24.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:24.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:24.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:24.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:24.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:24.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:24.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:24.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:24.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:24.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:24.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:24.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:24.289 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:24.289 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:24.289 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:24.289 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:24.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:24.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:24.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:24.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:24.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:24.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:24.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:24.411 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:24.411 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:24.411 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:24.411 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:24.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:24.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-6 2024-04-25T09:18:24.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-6 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-6 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:24.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:24.537 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-6 2024-04-25T09:18:24.537 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:24.537 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-6 2024-04-25T09:18:24.537 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-6 2024-04-25T09:18:24.537 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:24.952 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:24.953 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:24.953 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:24.953 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:24.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:24.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:24.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:24.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:24.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:24.953 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:24.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:24.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:24.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:24.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:24.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:24.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:24.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:24.956 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:24.956 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:24.956 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:24.956 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:24.956 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:24.956 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:24.956 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:24.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:25.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:25.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:25.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:25.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:25.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:25.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:25.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:25.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:25.067 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:25.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:25.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:25.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-6 2024-04-25T09:18:25.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:25.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:25.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:25.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:25.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:25.187 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:25.187 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:25.187 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:25.188 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:25.188 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:25.188 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:25.189 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:25.189 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:25.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:25.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:25.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-6 2024-04-25T09:18:25.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:25.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:25.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-6 2024-04-25T09:18:25.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:25.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:25.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:25.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-6 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:25.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:25.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:25.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:25.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:25.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:25.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:25.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:25.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:25.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T09:18:25.399 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:25.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:25.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:25.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:25.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:25.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:25.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:25.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:25.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:25.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:25.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:25.792 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:25.792 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:25.792 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:25.792 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:25.792 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:25.792 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:25.792 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:25.793 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:25.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:25.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:25.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:25.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:25.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:25.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:25.909 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:25.909 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:25.909 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:25.910 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:25.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:25.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-7 2024-04-25T09:18:26.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-7 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:26.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:26.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:26.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:26.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:26.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:26.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:26.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-7 2024-04-25T09:18:26.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:26.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:26.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:26.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:26.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:26.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-7 2024-04-25T09:18:26.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:26.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-7 2024-04-25T09:18:26.030 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-7 2024-04-25T09:18:26.031 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:26.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:26.442 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:26.442 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:26.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:26.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:26.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:26.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:26.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:26.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:26.444 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:26.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:26.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:26.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:26.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:26.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:26.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:26.445 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:26.445 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:26.445 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:26.445 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:26.445 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:26.445 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:26.446 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:26.446 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:26.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:26.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:26.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:26.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:26.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:26.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:26.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:26.556 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:26.556 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:26.556 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:26.557 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:26.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:26.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-7 2024-04-25T09:18:26.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:26.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:26.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:26.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:26.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:26.675 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:26.675 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:26.675 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:26.676 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:26.676 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:26.676 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:26.676 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:26.676 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:26.676 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:26.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:26.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-7 2024-04-25T09:18:26.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:26.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:26.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-7 2024-04-25T09:18:26.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:26.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:26.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:26.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-7 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:26.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:26.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:26.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:26.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:26.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:26.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T09:18:26.849 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:27.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:27.255 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:27.255 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:27.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:27.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:27.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:27.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:27.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:27.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:27.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:27.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:27.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:27.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:27.258 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:27.259 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:27.259 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:27.259 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:27.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:27.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:27.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:27.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:27.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:27.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:27.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:27.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:27.379 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:27.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:27.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:27.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-8 2024-04-25T09:18:27.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-8 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-8 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:27.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:27.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:27.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:27.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-8 2024-04-25T09:18:27.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:27.503 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-8 2024-04-25T09:18:27.503 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-8 2024-04-25T09:18:27.503 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:27.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:27.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:27.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:27.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:27.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:27.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:27.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:27.935 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:27.935 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:27.935 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:28.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:28.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:28.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:28.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:28.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:28.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:28.044 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:28.044 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:28.044 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:28.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:28.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:28.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-8 2024-04-25T09:18:28.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:28.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:28.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:28.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:28.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:28.155 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:28.155 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:28.155 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:28.155 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:28.155 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:28.155 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:28.155 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:28.155 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:28.155 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:28.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:28.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-8 2024-04-25T09:18:28.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:28.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:28.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-8 2024-04-25T09:18:28.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:28.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:28.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:28.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-8 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:28.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:28.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:28.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:28.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T09:18:28.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:28.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T09:18:28.315 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T09:18:28.315 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:28.699 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:28.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:28.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:28.700 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:28.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:28.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:28.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:28.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:28.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:28.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:28.702 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:28.703 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:28.703 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:28.703 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:28.703 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:28.703 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:28.703 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:28.703 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:28.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:28.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:28.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:28.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:28.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:28.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:28.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:28.820 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:28.820 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:28.820 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:28.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:28.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:28.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-9 2024-04-25T09:18:28.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-9 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:28.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:28.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-9 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-9 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:28.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-9 2024-04-25T09:18:28.931 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-9 2024-04-25T09:18:28.931 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:29.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:29.375 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:29.375 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:29.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:29.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:29.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:29.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:29.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:29.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:29.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:29.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:29.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:29.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:29.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:29.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:29.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:29.378 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:29.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:29.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:29.378 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:29.379 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:29.379 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:29.379 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:29.379 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:29.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:29.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:29.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:29.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:29.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:29.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:29.488 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:29.489 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:29.489 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:29.489 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:29.489 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:29.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:29.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-9 2024-04-25T09:18:29.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:29.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:29.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:29.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:29.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:29.605 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:29.605 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:29.605 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:29.605 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:29.605 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:29.606 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:29.606 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:29.606 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:29.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:29.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:29.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-9 2024-04-25T09:18:29.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:29.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:29.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-9 2024-04-25T09:18:29.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:29.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:29.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:29.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-9 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:29.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:29.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:29.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T09:18:29.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:29.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T09:18:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T09:18:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:30.191 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:30.192 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:30.192 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:30.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:30.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:30.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:30.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:30.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:30.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:30.192 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:30.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:30.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:30.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:30.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:30.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:30.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:30.194 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:30.194 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:30.194 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:30.194 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:30.195 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:30.195 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:30.195 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:30.195 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:30.195 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:30.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:30.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:30.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:30.313 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:30.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:30.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:30.314 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:30.314 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:30.314 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:30.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:30.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:30.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-10 2024-04-25T09:18:30.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-10 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:30.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-10 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-10 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:30.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-10 2024-04-25T09:18:30.437 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-10 2024-04-25T09:18:30.437 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:30.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:30.830 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:30.830 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:30.830 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:30.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:30.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:30.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:30.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:30.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:30.831 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:30.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:30.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:30.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:30.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:30.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:30.832 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:30.833 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:30.833 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:30.833 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:30.833 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:30.833 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:30.834 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:30.834 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:30.834 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:30.834 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:30.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:30.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:30.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:30.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:30.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:30.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:30.946 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:30.946 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:30.946 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:30.946 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:30.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:30.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-10 2024-04-25T09:18:31.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:31.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:31.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:31.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:31.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:31.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:31.063 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:31.063 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:31.064 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:31.064 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:31.064 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:31.064 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:31.065 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:31.065 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:31.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:31.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-10 2024-04-25T09:18:31.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:31.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:31.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-10 2024-04-25T09:18:31.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:31.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:31.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:31.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-10 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:31.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:31.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:31.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:222: TEST_rados_repair_warning: for i in $(seq 1 $OBJS) 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:224: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T09:18:31.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:31.248 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T09:18:31.248 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T09:18:31.248 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=3 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:31.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/3/type 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 3 bluestore_debug_inject_read_err true 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:31.692 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:31.692 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:31.692 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:31.692 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:31.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:31.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:31.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:31.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:31.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.3 2024-04-25T09:18:31.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:31.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:31.803 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:31.803 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:31.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:31.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.3.asok injectdataerr pool-rep obj-warn-11 2024-04-25T09:18:31.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:225: TEST_rados_repair_warning: inject_eio rep data pool-rep obj-warn-11 td/osd-rep-recov-eio.sh 1 2024-04-25T09:18:31.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:18:31.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:18:31.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:18:31.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:18:31.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=pool-rep 2024-04-25T09:18:31.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj-warn-11 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds pool-rep obj-warn-11 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=pool-rep 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj-warn-11 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map pool-rep obj-warn-11 2024-04-25T09:18:31.919 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:18:32.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=3 2024-04-25T09:18:32.314 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:32.314 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:18:32.314 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 3 1 0 2024-04-25T09:18:32.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('3' '1' '0') 2024-04-25T09:18:32.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:18:32.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:18:32.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:18:32.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:18:32.315 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:18:32.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:18:32.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:18:32.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:32.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:18:32.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:18:32.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:18:32.317 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:32.317 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:18:32.317 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:32.317 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:32.317 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:32.318 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:32.318 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:32.318 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:32.318 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:18:32.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:32.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:18:32.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:18:32.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:18:32.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:18:32.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:18:32.425 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:32.425 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:32.425 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:32.425 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:18:32.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:18:32.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr pool-rep obj-warn-11 2024-04-25T09:18:32.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:227: TEST_rados_repair_warning: set_config osd 3 osd_debug_feed_pullee 1 2024-04-25T09:18:32.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:18:32.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=3 2024-04-25T09:18:32.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=osd_debug_feed_pullee 2024-04-25T09:18:32.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=1 2024-04-25T09:18:32.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:18:32.533 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.3 2024-04-25T09:18:32.533 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.3 2024-04-25T09:18:32.533 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.3 ']' 2024-04-25T09:18:32.534 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:32.534 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:32.534 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:32.535 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.3.asok 2024-04-25T09:18:32.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.3.asok config set osd_debug_feed_pullee 1 2024-04-25T09:18:32.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:18:32.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:228: TEST_rados_repair_warning: rados_get td/osd-rep-recov-eio.sh pool-rep obj-warn-11 2024-04-25T09:18:32.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:80: rados_get: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:32.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:81: rados_get: local poolname=pool-rep 2024-04-25T09:18:32.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:82: rados_get: local objname=obj-warn-11 2024-04-25T09:18:32.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:83: rados_get: local expect=ok 2024-04-25T09:18:32.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:88: rados_get: '[' ok = fail ']' 2024-04-25T09:18:32.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:96: rados_get: '[' ok = hang ']' 2024-04-25T09:18:32.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:105: rados_get: rados --pool pool-rep get obj-warn-11 td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:32.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:106: rados_get: diff td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:32.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:107: rados_get: rm td/osd-rep-recov-eio.sh/COPY 2024-04-25T09:18:32.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:231: TEST_rados_repair_warning: wait_for_clean 2024-04-25T09:18:32.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:18:32.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:18:32.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:18:32.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:18:32.696 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:18:32.697 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:18:32.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:18:32.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:18:32.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:18:32.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:18:32.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:18:32.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:18:32.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:18:32.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:18:32.772 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:18:33.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:18:33.195 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:33.195 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:18:33.195 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:18:33.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:18:33.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:33.196 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:18:33.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803827 2024-04-25T09:18:33.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803827 2024-04-25T09:18:33.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803827' 2024-04-25T09:18:33.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:33.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:18:33.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607593 2024-04-25T09:18:33.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607593 2024-04-25T09:18:33.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803827 1-51539607593' 2024-04-25T09:18:33.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:33.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:18:33.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378659 2024-04-25T09:18:33.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378659 2024-04-25T09:18:33.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803827 1-51539607593 2-81604378659' 2024-04-25T09:18:33.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:33.719 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:18:33.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=120259084318 2024-04-25T09:18:33.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 120259084318 2024-04-25T09:18:33.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803827 1-51539607593 2-81604378659 3-120259084318' 2024-04-25T09:18:33.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:33.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803827 2024-04-25T09:18:33.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:33.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:18:33.869 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803827 2024-04-25T09:18:33.869 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:33.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803827 2024-04-25T09:18:33.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803827' 2024-04-25T09:18:33.870 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803827 2024-04-25T09:18:33.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:34.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803825 -lt 25769803827 2024-04-25T09:18:34.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:18:35.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:18:35.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:35.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803827 -lt 25769803827 2024-04-25T09:18:35.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:35.727 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607593 2024-04-25T09:18:35.727 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:35.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:18:35.728 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607593 2024-04-25T09:18:35.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:35.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607593 2024-04-25T09:18:35.730 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 51539607593 2024-04-25T09:18:35.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607593' 2024-04-25T09:18:35.731 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:18:36.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607594 -lt 51539607593 2024-04-25T09:18:36.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:36.155 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-81604378659 2024-04-25T09:18:36.155 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:36.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:18:36.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-81604378659 2024-04-25T09:18:36.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:36.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378659 2024-04-25T09:18:36.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 81604378659' 2024-04-25T09:18:36.159 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 81604378659 2024-04-25T09:18:36.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:18:36.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378660 -lt 81604378659 2024-04-25T09:18:36.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:36.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-120259084318 2024-04-25T09:18:36.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:36.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:18:36.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-120259084318 2024-04-25T09:18:36.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:36.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=120259084318 2024-04-25T09:18:36.586 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 120259084318 2024-04-25T09:18:36.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 120259084318' 2024-04-25T09:18:36.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:18:37.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 120259084319 -lt 120259084318 2024-04-25T09:18:37.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:18:37.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:18:37.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:18:37.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:18:37.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:18:37.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:18:37.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:18:37.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:18:37.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:18:37.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:18:37.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:18:38.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:18:38.023 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:18:38.023 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:18:38.023 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:18:38.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:18:38.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:18:38.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:18:38.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: ceph pg 2.0 query 2024-04-25T09:18:38.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: jq .info.stats.stat_sum.num_objects_repaired 2024-04-25T09:18:38.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:232: TEST_rados_repair_warning: COUNT=22 2024-04-25T09:18:38.764 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: expr 11 '*' 2 2024-04-25T09:18:38.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:233: TEST_rados_repair_warning: test 22 = 22 2024-04-25T09:18:38.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:234: TEST_rados_repair_warning: flush_pg_stats 2024-04-25T09:18:38.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:18:38.766 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:18:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:18:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:18:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:18:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:3' 2024-04-25T09:18:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:18:39.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:39.190 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:18:39.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803830 2024-04-25T09:18:39.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803830 2024-04-25T09:18:39.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830' 2024-04-25T09:18:39.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:39.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:18:39.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607596 2024-04-25T09:18:39.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607596 2024-04-25T09:18:39.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830 1-51539607596' 2024-04-25T09:18:39.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:39.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:18:39.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=81604378663 2024-04-25T09:18:39.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 81604378663 2024-04-25T09:18:39.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830 1-51539607596 2-81604378663' 2024-04-25T09:18:39.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:18:39.660 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:18:39.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=120259084322 2024-04-25T09:18:39.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 120259084322 2024-04-25T09:18:39.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803830 1-51539607596 2-81604378663 3-120259084322' 2024-04-25T09:18:39.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:39.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803830 2024-04-25T09:18:39.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:39.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:18:39.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803830 2024-04-25T09:18:39.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:39.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803830 2024-04-25T09:18:39.819 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803830 2024-04-25T09:18:39.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803830' 2024-04-25T09:18:39.820 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:40.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803828 -lt 25769803830 2024-04-25T09:18:40.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:18:41.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:18:41.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:18:41.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803830 -lt 25769803830 2024-04-25T09:18:41.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:41.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607596 2024-04-25T09:18:41.651 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:41.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:18:41.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:41.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607596 2024-04-25T09:18:41.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607596 2024-04-25T09:18:41.654 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 51539607596 2024-04-25T09:18:41.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607596' 2024-04-25T09:18:41.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:18:42.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607597 -lt 51539607596 2024-04-25T09:18:42.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:42.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:42.111 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-81604378663 2024-04-25T09:18:42.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:18:42.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-81604378663 2024-04-25T09:18:42.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:42.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=81604378663 2024-04-25T09:18:42.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 81604378663' 2024-04-25T09:18:42.114 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 81604378663 2024-04-25T09:18:42.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:18:42.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 81604378663 -lt 81604378663 2024-04-25T09:18:42.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:18:42.560 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-120259084322 2024-04-25T09:18:42.560 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:18:42.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:18:42.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-120259084322 2024-04-25T09:18:42.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:18:42.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=120259084322 2024-04-25T09:18:42.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 120259084322' 2024-04-25T09:18:42.565 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 120259084322 2024-04-25T09:18:42.565 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:18:42.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 120259084322 -lt 120259084322 2024-04-25T09:18:42.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: ceph pg dump --format=json-pretty 2024-04-25T09:18:42.970 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: jq .pg_map.osd_stats_sum.num_shards_repaired 2024-04-25T09:18:43.455 INFO:tasks.workunit.client.0.smithi134.stderr:dumped all 2024-04-25T09:18:43.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:235: TEST_rados_repair_warning: COUNT=33 2024-04-25T09:18:43.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: expr 11 '*' 3 2024-04-25T09:18:43.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:236: TEST_rados_repair_warning: test 33 = 33 2024-04-25T09:18:43.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:240: TEST_rados_repair_warning: CHECKTIME=10 2024-04-25T09:18:43.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:241: TEST_rados_repair_warning: LOOPS=0 2024-04-25T09:18:43.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:242: TEST_rados_repair_warning: true 2024-04-25T09:18:43.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:244: TEST_rados_repair_warning: sleep 1 2024-04-25T09:18:44.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: ceph health 2024-04-25T09:18:44.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:245: TEST_rados_repair_warning: grep -q 'Too many repaired reads on 2 OSDs' 2024-04-25T09:18:44.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:247: TEST_rados_repair_warning: break 2024-04-25T09:18:44.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: ceph health detail 2024-04-25T09:18:44.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: expr 11 '*' 2 2024-04-25T09:18:44.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:256: TEST_rados_repair_warning: grep -q 'osd.3 had 22 reads repaired' 2024-04-25T09:18:45.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: ceph health detail 2024-04-25T09:18:45.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:257: TEST_rados_repair_warning: grep -q 'osd.1 had 11 reads repaired' 2024-04-25T09:18:46.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:259: TEST_rados_repair_warning: delete_pool pool-rep 2024-04-25T09:18:46.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=pool-rep 2024-04-25T09:18:46.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete pool-rep pool-rep --yes-i-really-really-mean-it 2024-04-25T09:18:46.593 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'pool-rep' does not exist 2024-04-25T09:18:46.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:18:46.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:18:46.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:18:46.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:18:46.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:18:46.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:18:46.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:18:46.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:18:46.722 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:18:46.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:18:46.723 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:18:46.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:18:46.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:18:46.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:18:46.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:46.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:18:46.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:18:46.725 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:46.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:18:46.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:18:46.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:18:46.752 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:46.753 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:46.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.390086 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:18:46.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:18:46.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:18:46.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:18:46.756 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:18:46.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:18:46.758 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:18:46.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:18:46.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:18:46.759 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:18:46.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:46.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:18:46.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:18:46.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:18:46.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:18:46.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:18:46.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:18:46.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:46.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:46.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.390086 2024-04-25T09:18:46.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:18:46.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:18:46.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:18:46.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:46.765 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:46.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.390086 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:18:46.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:18:46.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:18:47.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:18:47.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:47.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:47.254 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:47.254 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:47.254 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:47.254 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:18:47.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:18:47.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:18:47.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:18:47.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:18:47.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:18:47.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:18:47.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:18:47.500 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:18:47.500 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:18:47.501 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:18:47.501 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:47.501 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:47.502 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:47.502 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-mon.a.asok 2024-04-25T09:18:47.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:18:47.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.390086/ceph-mon.a.asok config get fsid 2024-04-25T09:18:47.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:18:47.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:18:47.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:18:47.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:18:47.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:18:47.622 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:18:47.622 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:18:47.622 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:18:47.622 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:18:47.622 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:47.622 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:47.622 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-mon.a.asok 2024-04-25T09:18:47.622 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:18:47.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.390086/ceph-mon.a.asok config get mon_host 2024-04-25T09:18:47.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:18:47.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:47.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:18:47.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:18:47.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:18:47.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:18:47.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:18:48.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:18:48.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:48.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:48.656 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:48.656 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:48.656 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:48.656 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:18:48.657 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:18:48.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:18:48.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:18:49.009 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T09:18:49.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_backfill_unfound td/osd-rep-recov-eio.sh 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:264: TEST_rep_backfill_unfound: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:265: TEST_rep_backfill_unfound: local objname=myobject 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:266: TEST_rep_backfill_unfound: local lastobj=300 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:268: TEST_rep_backfill_unfound: local testobj=obj250 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:270: TEST_rep_backfill_unfound: export CEPH_ARGS 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:271: TEST_rep_backfill_unfound: CEPH_ARGS+=' --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:272: TEST_rep_backfill_unfound: setup_osds 3 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:18:49.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:18:49.028 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-04-25T09:18:49.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-04-25T09:18:49.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:18:49.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:18:49.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:18:49.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:18:49.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:18:49.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:18:49.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:18:49.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:18:49.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:18:49.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:18:49.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d093fc74-874f-4d42-a952-1a811aaed8c1 2024-04-25T09:18:49.034 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 d093fc74-874f-4d42-a952-1a811aaed8c1 2024-04-25T09:18:49.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d093fc74-874f-4d42-a952-1a811aaed8c1' 2024-04-25T09:18:49.035 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:18:49.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD5Hypm6rA/BBAAShzhyk+qFl8a6QtfKC1Q+A== 2024-04-25T09:18:49.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD5Hypm6rA/BBAAShzhyk+qFl8a6QtfKC1Q+A=="}' 2024-04-25T09:18:49.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d093fc74-874f-4d42-a952-1a811aaed8c1 -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:18:49.361 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:18:49.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:18:49.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD5Hypm6rA/BBAAShzhyk+qFl8a6QtfKC1Q+A== --osd-uuid d093fc74-874f-4d42-a952-1a811aaed8c1 2024-04-25T09:18:49.409 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:49.421+0000 7f71e799b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:49.429 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:49.441+0000 7f71e799b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:49.430 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:49.445+0000 7f71e799b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:49.431 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:49.445+0000 7f71e799b7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:18:49.431 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:49.445+0000 7f71e799b7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:18:49.431 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:49.445+0000 7f71e799b7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:18:49.469 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:49.481+0000 7f71e799b7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:18:55.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:18:55.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:18:55.717 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:18:55.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:18:55.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:18:56.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:18:56.262 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:18:56.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:18:56.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:18:56.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:18:56.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:18:56.295 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:56.309+0000 7f3ad99187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:56.311 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:56.309+0000 7f3ad99187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:56.312 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:56.313+0000 7f3ad99187c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:18:56.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:18:56.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:18:56.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:18:56.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:18:56.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:18:56.709 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:18:56.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:56.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:18:56.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:56.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:56.856 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:56.869+0000 7f3ad99187c0 -1 Falling back to public interface 2024-04-25T09:18:57.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:58.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:58.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:58.147 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:18:58.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:18:58.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:58.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:18:58.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:18:59.542 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:18:59.553+0000 7f3ad99187c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:18:59.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:18:59.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:18:59.559 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:18:59.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:18:59.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:18:59.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:00.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:01.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:01.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:01.252 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:19:01.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:19:01.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:01.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:01.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:02.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:02.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:02.708 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:19:02.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:19:02.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:02.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:03.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:04.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:04.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:04.156 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:19:04.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:19:04.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:04.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:04.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:05.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:05.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:05.788 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:19:05.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:19:05.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:05.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:06.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:07.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:07.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:07.217 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:19:07.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:19:07.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:07.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:07.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:08.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:08.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:08.776 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:19:08.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:19:08.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:08.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:09.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:10.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:10.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:10.236 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:19:10.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:19:10.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:10.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:10.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:11.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:11.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:11.744 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:19:11.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:19:11.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:11.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:12.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:13.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:13.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:13.196 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:19:13.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:19:13.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:13.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:14.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:15.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:15.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:15.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:19:15.058 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:19:15.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:15.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:15.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:16.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:16.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:16.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:19:16.967 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:19:16.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:16.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:17.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:18.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:18.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:18.784 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:19:18.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:19:18.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:18.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:19.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:20.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:20.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:20.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:19:20.548 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:19:20.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:20.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:21.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:22.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:22.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:22.420 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:19:22.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:19:22.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:22.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:23.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:24.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:24.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:24.523 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:19:24.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:19:24.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:24.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:25.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:26.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:26.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:26.037 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:19:26.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:19:26.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:26.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:19:26.469 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1662437489,v1:127.0.0.1:6803/1662437489] [v2:127.0.0.1:6804/1662437489,v1:127.0.0.1:6805/1662437489] exists,up d093fc74-874f-4d42-a952-1a811aaed8c1 2024-04-25T09:19:26.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:19:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:19:26.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:19:26.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:19:26.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=08e4edc9-af16-4570-bfc4-2a8769debb1f 2024-04-25T09:19:26.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 08e4edc9-af16-4570-bfc4-2a8769debb1f' 2024-04-25T09:19:26.477 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 08e4edc9-af16-4570-bfc4-2a8769debb1f 2024-04-25T09:19:26.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:19:26.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAeICpmGYNHHhAAa+64Z4IWMCfeZCrTbcArRA== 2024-04-25T09:19:26.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAeICpmGYNHHhAAa+64Z4IWMCfeZCrTbcArRA=="}' 2024-04-25T09:19:26.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 08e4edc9-af16-4570-bfc4-2a8769debb1f -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:19:27.016 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:19:27.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:19:27.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAeICpmGYNHHhAAa+64Z4IWMCfeZCrTbcArRA== --osd-uuid 08e4edc9-af16-4570-bfc4-2a8769debb1f 2024-04-25T09:19:27.072 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:27.084+0000 7f7a998df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:27.101 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:27.112+0000 7f7a998df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:27.103 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:27.116+0000 7f7a998df7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:27.104 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:27.116+0000 7f7a998df7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:19:27.104 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:27.116+0000 7f7a998df7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:19:27.104 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:27.116+0000 7f7a998df7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:19:27.133 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:27.144+0000 7f7a998df7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:19:33.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:19:33.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:19:33.821 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:19:33.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:19:33.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:19:34.401 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:19:34.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:19:34.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:19:34.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:19:34.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:19:34.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:19:34.435 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:34.448+0000 7fe9b67a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:34.472 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:34.484+0000 7fe9b67a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:34.473 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:34.484+0000 7fe9b67a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:19:34.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:19:34.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:19:34.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:19:34.842 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:19:34.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:19:34.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:19:34.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:34.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:19:34.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:34.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:35.032 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:35.044+0000 7fe9b67a87c0 -1 Falling back to public interface 2024-04-25T09:19:35.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:36.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:36.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:36.288 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:19:36.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:19:36.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:36.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:36.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:37.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:37.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:37.735 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:19:37.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:19:37.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:37.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:37.849 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:19:37.860+0000 7fe9b67a87c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:19:38.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:39.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:39.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:39.340 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:19:39.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:19:39.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:39.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:39.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:40.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:40.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:40.985 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:19:40.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:19:40.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:40.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:41.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:42.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:42.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:42.426 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:19:42.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:19:42.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:42.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:43.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:44.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:44.027 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:19:44.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:44.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:19:44.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:44.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:44.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:45.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:45.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:45.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:19:45.620 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:19:45.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:45.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:46.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:47.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:47.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:47.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:19:47.053 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:19:47.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:47.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:47.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:48.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:48.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:48.471 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:19:48.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:19:48.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:48.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:48.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:49.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:49.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:49.997 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:19:49.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:19:49.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:49.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:50.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:51.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:51.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:51.445 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:19:51.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:19:51.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:51.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:51.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:52.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:52.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:52.990 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:19:52.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:19:52.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:52.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:53.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:54.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:54.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:54.405 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:19:54.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:19:54.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:54.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:55.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:56.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:56.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:56.173 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:19:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:19:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:57.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:19:58.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:19:58.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:19:58.348 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:19:58.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:19:58.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:19:58.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:19:59.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:00.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:00.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:00.361 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:20:00.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:20:00.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:00.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:01.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:02.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:02.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:02.351 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:20:02.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:20:02.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:02.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:03.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:04.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:04.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:04.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:20:04.199 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:20:04.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:04.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:04.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:05.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:05.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:05.807 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T09:20:05.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:20:05.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:05.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:20:06.337 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 12 up_thru 12 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/451295025,v1:127.0.0.1:6811/451295025] [v2:127.0.0.1:6812/451295025,v1:127.0.0.1:6813/451295025] exists,up 08e4edc9-af16-4570-bfc4-2a8769debb1f 2024-04-25T09:20:06.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:20:06.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:20:06.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:20:06.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:20:06.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:20:06.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:20:06.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:20:06.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:20:06.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:20:06.342 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:20:06.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:20:06.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:20:06.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:20:06.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:20:06.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:20:06.344 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:20:06.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ad089f9-ca90-4ae7-93df-92a1230bd42a 2024-04-25T09:20:06.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6ad089f9-ca90-4ae7-93df-92a1230bd42a' 2024-04-25T09:20:06.345 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 6ad089f9-ca90-4ae7-93df-92a1230bd42a 2024-04-25T09:20:06.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:20:06.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGICpm3aSrFhAA/XshBDel8PyECGZRmZzvRw== 2024-04-25T09:20:06.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGICpm3aSrFhAA/XshBDel8PyECGZRmZzvRw=="}' 2024-04-25T09:20:06.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ad089f9-ca90-4ae7-93df-92a1230bd42a -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:20:06.958 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:20:06.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:20:06.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGICpm3aSrFhAA/XshBDel8PyECGZRmZzvRw== --osd-uuid 6ad089f9-ca90-4ae7-93df-92a1230bd42a 2024-04-25T09:20:07.004 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:07.015+0000 7f58f88be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:07.006 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:07.019+0000 7f58f88be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:07.007 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:07.019+0000 7f58f88be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:07.008 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:07.019+0000 7f58f88be7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:20:07.008 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:07.019+0000 7f58f88be7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:20:07.008 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:07.019+0000 7f58f88be7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:20:07.132 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:07.143+0000 7f58f88be7c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:20:14.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:20:14.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:20:14.148 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:20:14.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:20:14.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:20:14.700 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:20:14.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:20:14.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:20:14.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:20:14.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:20:14.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:20:14.736 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:14.747+0000 7f1bf5a927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:14.738 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:14.751+0000 7f1bf5a927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:14.739 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:14.751+0000 7f1bf5a927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:20:15.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:20:15.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:20:15.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:20:15.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:20:15.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:20:15.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:15.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:20:15.101 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:20:15.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:15.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:15.304 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:15.315+0000 7f1bf5a927c0 -1 Falling back to public interface 2024-04-25T09:20:15.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:16.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:16.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:16.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:20:16.494 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:20:16.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:16.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:16.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:17.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:17.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:17.966 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:20:17.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:20:17.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:17.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:18.017 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:20:18.027+0000 7f1bf5a927c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:20:18.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:19.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:19.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:19.430 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:20:19.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:20:19.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:19.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:20.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:21.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:21.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:21.148 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:20:21.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:20:21.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:21.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:21.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:22.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:22.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:22.632 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:20:22.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:20:22.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:22.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:23.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:24.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:24.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:24.102 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:20:24.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:20:24.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:24.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:24.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:25.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:25.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:25.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:20:25.524 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:20:25.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:25.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:26.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:27.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:27.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:27.111 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:20:27.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:20:27.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:27.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:27.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:28.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:28.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:28.650 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:20:28.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:20:28.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:28.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:29.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:30.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:30.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:30.167 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:20:30.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:20:30.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:30.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:30.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:31.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:31.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:31.676 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:20:31.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:20:31.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:31.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:32.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:33.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:33.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:33.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:20:33.146 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:20:33.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:33.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:34.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:35.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:35.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:35.071 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:20:35.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:20:35.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:35.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:35.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:36.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:36.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:36.794 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:20:36.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:20:36.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:36.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:37.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:38.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:38.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:38.644 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:20:38.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:20:38.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:38.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:39.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:40.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:40.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:40.670 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:20:40.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:20:40.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:40.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:41.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:42.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:42.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:42.969 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:20:42.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:20:42.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:42.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:43.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:20:44.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:20:44.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:20:44.973 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:20:44.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:20:44.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:20:44.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:20:45.434 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3127132104,v1:127.0.0.1:6819/3127132104] [v2:127.0.0.1:6820/3127132104,v1:127.0.0.1:6821/3127132104] exists,up 6ad089f9-ca90-4ae7-93df-92a1230bd42a 2024-04-25T09:20:45.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:20:45.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:20:45.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:20:45.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:20:45.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:20:45.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:20:45.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:20:45.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:20:45.436 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:20:45.436 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:20:45.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:20:45.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:20:45.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:20:45.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:20:45.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:20:45.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:20:45.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:20:45.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:20:45.529 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:20:46.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:20:46.132 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:20:46.132 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:20:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:20:46.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:20:46.133 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:20:46.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803798 2024-04-25T09:20:46.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803798 2024-04-25T09:20:46.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803798' 2024-04-25T09:20:46.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:20:46.324 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:20:46.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607570 2024-04-25T09:20:46.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607570 2024-04-25T09:20:46.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803798 1-51539607570' 2024-04-25T09:20:46.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:20:46.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:20:46.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345933 2024-04-25T09:20:46.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345933 2024-04-25T09:20:46.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803798 1-51539607570 2-85899345933' 2024-04-25T09:20:46.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:20:46.638 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803798 2024-04-25T09:20:46.638 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:20:46.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:20:46.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803798 2024-04-25T09:20:46.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:20:46.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803798 2024-04-25T09:20:46.641 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803798 2024-04-25T09:20:46.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803798' 2024-04-25T09:20:46.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:20:47.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803795 -lt 25769803798 2024-04-25T09:20:47.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:20:48.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:20:48.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:20:48.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803798 -lt 25769803798 2024-04-25T09:20:48.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:20:48.480 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607570 2024-04-25T09:20:48.480 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:20:48.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:20:48.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607570 2024-04-25T09:20:48.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:20:48.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607570 2024-04-25T09:20:48.483 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 51539607570 2024-04-25T09:20:48.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607570' 2024-04-25T09:20:48.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:20:48.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607570 -lt 51539607570 2024-04-25T09:20:48.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:20:48.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345933 2024-04-25T09:20:48.930 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:20:48.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:20:48.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345933 2024-04-25T09:20:48.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:20:48.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345933 2024-04-25T09:20:48.933 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 85899345933 2024-04-25T09:20:48.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345933' 2024-04-25T09:20:48.934 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:20:49.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345933 -lt 85899345933 2024-04-25T09:20:49.347 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:20:49.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:20:49.348 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:20:49.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:20:49.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:20:49.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:20:49.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:20:49.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:20:49.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:20:49.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:20:49.891 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:20:50.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=5 2024-04-25T09:20:50.305 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:20:50.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:20:50.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:20:50.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 5 = 8 2024-04-25T09:20:50.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 5 '!=' -1 2024-04-25T09:20:50.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:20:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=5 2024-04-25T09:20:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:20:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:20:50.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:20:50.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:20:50.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:20:50.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:20:50.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:20:50.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:20:50.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:20:50.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:20:51.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:20:51.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:20:51.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:20:51.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:20:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:20:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:20:51.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:20:51.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:274: TEST_rep_backfill_unfound: local poolname=test-pool 2024-04-25T09:20:51.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:275: TEST_rep_backfill_unfound: create_pool test-pool 1 1 2024-04-25T09:20:51.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-04-25T09:20:52.453 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test-pool' already exists 2024-04-25T09:20:52.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:20:53.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:276: TEST_rep_backfill_unfound: wait_for_clean 2024-04-25T09:20:53.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:20:53.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:20:53.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:20:53.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:20:53.472 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:20:53.473 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:20:53.473 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:20:53.473 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:20:53.473 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:20:53.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:20:53.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:20:53.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:20:53.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:20:53.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:20:53.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:20:54.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:20:54.030 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:20:54.030 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:20:54.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:20:54.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:20:54.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:20:54.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803801 2024-04-25T09:20:54.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803801 2024-04-25T09:20:54.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801' 2024-04-25T09:20:54.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:20:54.268 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:20:54.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607573 2024-04-25T09:20:54.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607573 2024-04-25T09:20:54.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801 1-51539607573' 2024-04-25T09:20:54.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:20:54.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:20:54.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345936 2024-04-25T09:20:54.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345936 2024-04-25T09:20:54.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801 1-51539607573 2-85899345936' 2024-04-25T09:20:54.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:20:54.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803801 2024-04-25T09:20:54.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:20:54.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:20:54.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803801 2024-04-25T09:20:54.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:20:54.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803801 2024-04-25T09:20:54.599 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803801 2024-04-25T09:20:54.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2024-04-25T09:20:54.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:20:55.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803799 -lt 25769803801 2024-04-25T09:20:55.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:20:56.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:20:56.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:20:56.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803801 -lt 25769803801 2024-04-25T09:20:56.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:20:56.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:20:56.424 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607573 2024-04-25T09:20:56.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:20:56.426 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607573 2024-04-25T09:20:56.426 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:20:56.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607573 2024-04-25T09:20:56.427 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 51539607573 2024-04-25T09:20:56.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607573' 2024-04-25T09:20:56.428 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:20:56.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607573 -lt 51539607573 2024-04-25T09:20:56.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:20:56.912 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345936 2024-04-25T09:20:56.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:20:56.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:20:56.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345936 2024-04-25T09:20:56.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:20:56.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345936 2024-04-25T09:20:56.917 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 85899345936 2024-04-25T09:20:56.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345936' 2024-04-25T09:20:56.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:20:57.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345936 -lt 85899345936 2024-04-25T09:20:57.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:20:57.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:20:57.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:20:57.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:20:57.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:20:57.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:20:57.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:20:57.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:20:57.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:20:57.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:20:57.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:20:58.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:20:58.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:20:58.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:20:58.365 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:20:58.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:20:58.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:20:58.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:20:58.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:278: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T09:20:59.324 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:20:59.324 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:20:46.254916+0000 0'0 25:34 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:19:53.071060+0000 0 0 2024-04-25T09:20:59.324 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:20:46.255469+0000 0'0 25:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:36:12.574497+0000 0 0 2024-04-25T09:20:59.324 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:20:47.258020+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:06:44.278783+0000 0 0 2024-04-25T09:20:59.324 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:20:46.256924+0000 0'0 25:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:22:08.673095+0000 0 0 2024-04-25T09:20:59.324 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:20:54.108975+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:20:52.285243+0000 0'0 2024-04-25T09:20:52.285243+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:43:47.823196+0000 0 0 2024-04-25T09:20:59.324 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:20:46.257623+0000 0'0 25:34 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:13:27.496270+0000 0 0 2024-04-25T09:20:59.324 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:20:47.416736+0000 0'0 26:48 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:15:36.750394+0000 0 0 2024-04-25T09:20:59.324 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:20:47.416697+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:02:39.802985+0000 0 0 2024-04-25T09:20:59.325 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:20:46.257037+0000 0'0 25:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:00:53.663493+0000 0 0 2024-04-25T09:20:59.325 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T09:20:59.325 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:20:59.325 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T09:20:59.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:280: TEST_rep_backfill_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-04-25T09:20:59.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:20:59.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-04-25T09:20:59.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-04-25T09:20:59.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:20:59.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:20:59.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:20:59.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:20:59.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:20:59.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:20:59.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:20:59.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:20:59.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:20:59.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: get_osds test-pool myobject 2024-04-25T09:20:59.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:20:59.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-25T09:20:59.535 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-04-25T09:20:59.535 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:20:59.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:20:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:20:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:20:59.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-25T09:20:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: initial_osds=('2' '1' '0') 2024-04-25T09:20:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:282: TEST_rep_backfill_unfound: local -a initial_osds 2024-04-25T09:20:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:283: TEST_rep_backfill_unfound: local last_osd=0 2024-04-25T09:20:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:284: TEST_rep_backfill_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-04-25T09:20:59.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:20:59.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:20:59.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:20:59.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:20:59.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:21:00.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:21:00.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:285: TEST_rep_backfill_unfound: ceph osd down 0 2024-04-25T09:21:01.092 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T09:21:01.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:286: TEST_rep_backfill_unfound: ceph osd out 0 2024-04-25T09:21:01.715 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T09:21:01.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:288: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T09:21:02.189 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:21:02.189 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:21:00.302050+0000 0'0 26:37 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:21:02.189 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:21:00.302006+0000 0'0 26:37 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:21:02.189 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:21:00.231625+0000 0'0 26:19 [2,1] 2 [2,1] 2 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:21:02.190 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:21:00.302098+0000 0'0 26:37 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:21:02.190 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 1 0 0 0 0 4096 0 0 1 0 1 peering 2024-04-25T09:21:00.230892+0000 26'1 26:15 [2,1] 2 [2,1] 2 0'0 2024-04-25T09:20:52.285243+0000 0'0 2024-04-25T09:20:52.285243+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:21:02.190 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:21:00.302560+0000 0'0 26:37 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:21:02.190 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:20:45.470648+0000 0'0 27:36 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:21:02.190 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:21:00.232025+0000 0'0 26:19 [2,1] 2 [2,1] 2 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:21:02.190 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 peering 2024-04-25T09:21:00.302847+0000 0'0 26:37 [1,2] 1 [1,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:21:02.190 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T09:21:02.190 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:21:02.190 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T09:21:02.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:290: TEST_rep_backfill_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-04-25T09:21:02.206 INFO:tasks.workunit.client.0.smithi134.stderr:4+0 records in 2024-04-25T09:21:02.206 INFO:tasks.workunit.client.0.smithi134.stderr:4+0 records out 2024-04-25T09:21:02.206 INFO:tasks.workunit.client.0.smithi134.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000190569 s, 21.5 MB/s 2024-04-25T09:21:02.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: seq 1 300 2024-04-25T09:21:02.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:02.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:02.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:02.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:02.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:02.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:02.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:02.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:02.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:02.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:02.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:02.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:03.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:03.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:03.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:03.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:03.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:03.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:03.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:03.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:03.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:03.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:04.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:04.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:04.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:04.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:04.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:04.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:04.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:04.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:04.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:04.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:04.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:04.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:04.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:04.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:05.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:05.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:05.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:05.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:05.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:05.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:05.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:05.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:05.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:05.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:06.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:06.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:06.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:06.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:06.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:06.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:06.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:06.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:06.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:06.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:06.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:06.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:06.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:06.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:07.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:07.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:07.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:07.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:07.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:07.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:07.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:07.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:07.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:07.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:08.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:08.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:08.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:08.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:08.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:08.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:08.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:08.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:08.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:08.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:08.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:08.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:08.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:08.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:09.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:09.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:09.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:09.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:09.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:09.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:09.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:09.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:09.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:09.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:10.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:10.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:10.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:10.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:10.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:10.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:10.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:10.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:10.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:10.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:10.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:10.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:11.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:11.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:11.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:11.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:11.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:11.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:11.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:11.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:11.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:11.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:11.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:11.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:12.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:12.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:12.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:12.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:12.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:12.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:12.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:12.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:12.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:12.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:12.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:12.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:12.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:12.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:13.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:13.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:13.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:13.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:13.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:13.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:13.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:13.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:13.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:13.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:14.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:14.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:14.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:14.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:14.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:14.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:14.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:14.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:14.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:14.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:14.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:14.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:15.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:15.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:15.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:15.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:15.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:15.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:15.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:15.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:15.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:15.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:15.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:15.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:16.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:16.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:16.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:16.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:16.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:16.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:16.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:16.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:16.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:16.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:16.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:16.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:16.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:16.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:17.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:17.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:17.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:17.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:17.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:17.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:17.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:17.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:17.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:17.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:18.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:18.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:18.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:18.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:18.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:18.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:18.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:18.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:18.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:18.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:18.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:18.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj101 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:18.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:18.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj102 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:19.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:19.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj103 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:19.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:19.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj104 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:19.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:19.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj105 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:19.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:19.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj106 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:19.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:19.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj107 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:19.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:19.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj108 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:20.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:20.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj109 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:20.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:20.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj110 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:20.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:20.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj111 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:20.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:20.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj112 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:20.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:20.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj113 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:20.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:20.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj114 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:21.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:21.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj115 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:21.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:21.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj116 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:21.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:21.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj117 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:21.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:21.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj118 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:21.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:21.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj119 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:21.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:21.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj120 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:22.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:22.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj121 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:22.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:22.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj122 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:22.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:22.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj123 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:22.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:22.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj124 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:22.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:22.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj125 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:22.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:22.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj126 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:22.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:22.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj127 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:23.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:23.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj128 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:23.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:23.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj129 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:23.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:23.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj130 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:23.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:23.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj131 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:23.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:23.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj132 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:23.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:23.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj133 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:24.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:24.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj134 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:24.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:24.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj135 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:24.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:24.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj136 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:24.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:24.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj137 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:24.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:24.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj138 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:24.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:24.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj139 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:25.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:25.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj140 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:25.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj141 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:25.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:25.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj142 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:25.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:25.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj143 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:25.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:25.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj144 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:25.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:25.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj145 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:26.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:26.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj146 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:26.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:26.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj147 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:26.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:26.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj148 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:26.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:26.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj149 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:26.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:26.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj150 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:26.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:26.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj151 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:27.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:27.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj152 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:27.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:27.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj153 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:27.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:27.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj154 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:27.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:27.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj155 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:27.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:27.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj156 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:27.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:27.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj157 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:28.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:28.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj158 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:28.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:28.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj159 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:28.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:28.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj160 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:28.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:28.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj161 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:28.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:28.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj162 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:28.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:28.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj163 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:29.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:29.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj164 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:29.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:29.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj165 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:29.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:29.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj166 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:29.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:29.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj167 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:29.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:29.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj168 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:29.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:29.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj169 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:30.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:30.038 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj170 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:30.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:30.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj171 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:30.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:30.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj172 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:30.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:30.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj173 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:30.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:30.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj174 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:30.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:30.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj175 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:30.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:30.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj176 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:31.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:31.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj177 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:31.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:31.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj178 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:31.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:31.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj179 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:31.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:31.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj180 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:31.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:31.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj181 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:31.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:31.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj182 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:32.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:32.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj183 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:32.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:32.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj184 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:32.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:32.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj185 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:32.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:32.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj186 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:32.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:32.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj187 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:32.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj188 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:33.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:33.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj189 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:33.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:33.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj190 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:33.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:33.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj191 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:33.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:33.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj192 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:33.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:33.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj193 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:33.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:33.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj194 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:34.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:34.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj195 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:34.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:34.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj196 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:34.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:34.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj197 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:34.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:34.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj198 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:34.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:34.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj199 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:34.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:34.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj200 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:35.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:35.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj201 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:35.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:35.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj202 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:35.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:35.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj203 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:35.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:35.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj204 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:35.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:35.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj205 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:35.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:35.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj206 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:36.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:36.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj207 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:36.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:36.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj208 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:36.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:36.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj209 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:36.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:36.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj210 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:36.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:36.715 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj211 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:36.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:36.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj212 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:37.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:37.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj213 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:37.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:37.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj214 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:37.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:37.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj215 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:37.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:37.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj216 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:37.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:37.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj217 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:37.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:37.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj218 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:38.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:38.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj219 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:38.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:38.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj220 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:38.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:38.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj221 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:38.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:38.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj222 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:38.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:38.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj223 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:38.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:38.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj224 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:38.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:38.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj225 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:39.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:39.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj226 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:39.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:39.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj227 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:39.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:39.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj228 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:39.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:39.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj229 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:40.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:40.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj230 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:40.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:40.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj231 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:40.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:40.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj232 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:40.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:40.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj233 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:40.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:40.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj234 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:40.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:40.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj235 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:41.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:41.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj236 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:41.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:41.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj237 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:41.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:41.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj238 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:41.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:41.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj239 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:41.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:41.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj240 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:41.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:41.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj241 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:42.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:42.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj242 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:42.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:42.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj243 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:42.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:42.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj244 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:42.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:42.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj245 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:42.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:42.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj246 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:42.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:42.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj247 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:43.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:43.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj248 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:43.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:43.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj249 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:43.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:43.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj250 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:43.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:43.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj251 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:43.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:43.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj252 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:43.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:43.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj253 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:44.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:44.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj254 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:44.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:44.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj255 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:44.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:44.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj256 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:44.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:44.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj257 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:44.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:44.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj258 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:44.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:44.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj259 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:45.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:45.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj260 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:45.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:45.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj261 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:45.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:45.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj262 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:45.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:45.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj263 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:45.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:45.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj264 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:45.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:45.868 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj265 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:46.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:46.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj266 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:46.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:46.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj267 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:46.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:46.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj268 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:46.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:46.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj269 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:46.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:46.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj270 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:46.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:46.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj271 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:47.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:47.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj272 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:47.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:47.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj273 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:47.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:47.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj274 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:47.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:47.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj275 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:47.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:47.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj276 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:47.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:47.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj277 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:48.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:48.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj278 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:48.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:48.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj279 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:48.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:48.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj280 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:48.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:48.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj281 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:48.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:48.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj282 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:48.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:48.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj283 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:49.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:49.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj284 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:49.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:49.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj285 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:49.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:49.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj286 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:49.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:49.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj287 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:49.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:49.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj288 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:49.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:49.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj289 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:50.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:50.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj290 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:50.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:50.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj291 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:50.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:50.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj292 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj293 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:50.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:50.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj294 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:50.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:50.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj295 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:50.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:50.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj296 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:51.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:51.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj297 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:51.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:51.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj298 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:51.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:51.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj299 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:51.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:291: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:21:51.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:293: TEST_rep_backfill_unfound: rados --pool test-pool put obj300 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:21:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:296: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 0 2024-04-25T09:21:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:21:51.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj250 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj250 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:21:51.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-25T09:21:51.828 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-04-25T09:21:51.828 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:21:52.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:21:52.257 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T09:21:52.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:21:52.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:21:52.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:21:52.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=2 2024-04-25T09:21:52.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:21:52.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:21:52.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-04-25T09:21:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:21:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-04-25T09:21:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:21:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-04-25T09:21:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:21:52.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:21:52.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:21:52.260 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-04-25T09:21:52.260 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:21:52.260 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:21:52.260 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:21:52.260 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:52.261 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:21:52.261 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.2.asok 2024-04-25T09:21:52.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:21:52.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:21:52.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:21:52.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.2 2024-04-25T09:21:52.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:21:52.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:21:52.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:21:52.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:21:52.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:52.384 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:21:52.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.2.asok 2024-04-25T09:21:52.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:21:52.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.2.asok injectdataerr test-pool obj250 2024-04-25T09:21:52.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:297: TEST_rep_backfill_unfound: inject_eio rep data test-pool obj250 td/osd-rep-recov-eio.sh 1 2024-04-25T09:21:52.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:21:52.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:21:52.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:21:52.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:21:52.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:21:52.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:21:52.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj250 2024-04-25T09:21:52.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:21:52.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:21:52.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:21:52.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:21:52.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:21:52.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj250 2024-04-25T09:21:52.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:21:52.505 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj250 2024-04-25T09:21:52.505 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj250 2024-04-25T09:21:52.505 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:21:52.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:21:52.924 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T09:21:52.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:21:52.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:21:52.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:21:52.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:21:52.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:21:52.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:21:52.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:21:52.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:21:52.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:21:52.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:21:52.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:21:52.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:21:52.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:21:52.926 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:21:52.926 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:21:52.926 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:21:52.926 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:21:52.926 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:21:52.927 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:52.927 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:21:52.927 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:21:52.927 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:21:53.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:21:53.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:21:53.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:21:53.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:21:53.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:21:53.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:21:53.049 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:21:53.049 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:53.049 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:21:53.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:21:53.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:21:53.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr test-pool obj250 2024-04-25T09:21:53.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:299: TEST_rep_backfill_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:21:53.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:21:53.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:21:53.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:21:53.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:21:53.173 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:21:53.173 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:21:53.173 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:21:53.173 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:21:53.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:21:53.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:21:53.175 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:21:53.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:21:53.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:21:53.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-04-25T09:21:53.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:21:53.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:21:53.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:21:53.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:21:53.212 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:21:53.225+0000 7f1ce701f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:53.215 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:21:53.229+0000 7f1ce701f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:53.216 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:21:53.229+0000 7f1ce701f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:21:53.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:21:53.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:21:53.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:21:53.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:21:53.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:21:53.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:53.611 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:21:53.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:21:53.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:53.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:53.764 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:21:53.777+0000 7f1ce701f7c0 -1 Falling back to public interface 2024-04-25T09:21:54.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:55.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:55.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:55.063 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:21:55.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:21:55.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:55.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:55.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:56.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:56.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:56.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:21:56.531 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:21:56.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:56.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:56.837 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:21:56.848+0000 7f1ce701f7c0 -1 osd.0 26 log_to_monitors true 2024-04-25T09:21:56.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:21:57.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:21:57.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:21:57.949 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:21:57.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:21:57.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:21:57.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:21:58.703 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up out weight 0 up_from 32 up_thru 21 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/2375529999,v1:127.0.0.1:6803/2375529999] [v2:127.0.0.1:6804/2375529999,v1:127.0.0.1:6805/2375529999] exists,up d093fc74-874f-4d42-a952-1a811aaed8c1 2024-04-25T09:21:58.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:21:58.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:21:58.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:21:58.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:300: TEST_rep_backfill_unfound: ceph osd in 0 2024-04-25T09:22:00.341 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already in. 2024-04-25T09:22:00.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:302: TEST_rep_backfill_unfound: sleep 15 2024-04-25T09:22:15.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: seq 1 360 2024-04-25T09:22:15.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:22:15.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:22:15.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:22:15.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:22:15.367 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:22:15.368 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:22:15.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:15.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:15.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:22:15.794 INFO:tasks.workunit.client.0.smithi134.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:15.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:22:15.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:22:15.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:22:16.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:22:16.796 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:22:16.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:22:16.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:22:16.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:22:16.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:22:17.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:17.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:17.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:22:17.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:22:17.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:22:17.249 INFO:tasks.workunit.client.0.smithi134.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:17.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:22:18.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:22:18.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:22:18.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:22:18.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:22:18.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:22:18.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:22:18.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:18.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:18.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:22:18.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:22:18.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:22:18.674 INFO:tasks.workunit.client.0.smithi134.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:18.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:22:19.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:22:19.676 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:22:19.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:22:19.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:22:19.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:22:19.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:22:20.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:20.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:20.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:22:20.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 1 = 0 ']' 2024-04-25T09:22:20.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:310: TEST_rep_backfill_unfound: echo 'active+undersized+degraded+remapped+backfilling ' 2024-04-25T09:22:20.139 INFO:tasks.workunit.client.0.smithi134.stdout:active+undersized+degraded+remapped+backfilling 2024-04-25T09:22:20.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:311: TEST_rep_backfill_unfound: sleep 1 2024-04-25T09:22:21.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:304: TEST_rep_backfill_unfound: for tmp in $(seq 1 360) 2024-04-25T09:22:21.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: get_state 2.0 2024-04-25T09:22:21.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:22:21.141 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:22:21.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:22:21.142 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:22:21.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:305: TEST_rep_backfill_unfound: state=active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:22:21.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: echo active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:22:21.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:306: TEST_rep_backfill_unfound: grep backfill_unfound 2024-04-25T09:22:21.571 INFO:tasks.workunit.client.0.smithi134.stdout:active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:22:21.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:307: TEST_rep_backfill_unfound: '[' 0 = 0 ']' 2024-04-25T09:22:21.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:308: TEST_rep_backfill_unfound: break 2024-04-25T09:22:21.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:314: TEST_rep_backfill_unfound: ceph pg dump pgs 2024-04-25T09:22:21.963 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:22:21.963 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:22:01.104367+0000 0'0 41:78 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:25:34.714411+0000 0 0 2024-04-25T09:22:21.963 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:22:02.193836+0000 0'0 41:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:31:24.732145+0000 0 0 2024-04-25T09:22:21.963 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:22:02.193141+0000 0'0 41:58 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:47:35.377140+0000 0 0 2024-04-25T09:22:21.963 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:22:02.192402+0000 0'0 41:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T13:26:43.410833+0000 0 0 2024-04-25T09:22:21.963 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 301 1 28 0 1 1232896 0 0 101 200 101 active+backfill_unfound+undersized+degraded+remapped 2024-04-25T09:22:20.392695+0000 30'301 41:895 [2,1,0] 2 [2,1] 2 0'0 2024-04-25T09:20:52.285243+0000 0'0 2024-04-25T09:20:52.285243+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:22:21.964 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:22:01.103500+0000 0'0 41:78 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:53:06.838391+0000 0 0 2024-04-25T09:22:21.964 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:22:01.510946+0000 0'0 41:66 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:23:35.171104+0000 0 0 2024-04-25T09:22:21.964 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:22:02.192321+0000 0'0 41:58 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T10:18:56.933555+0000 0 0 2024-04-25T09:22:21.964 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:22:02.099051+0000 0'0 41:76 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:18:48.893297+0000 0'0 2024-04-25T09:18:48.893297+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:59:28.852644+0000 0 0 2024-04-25T09:22:21.964 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T09:22:21.964 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:22:21.964 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T09:22:21.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: ceph pg 2.0 list_unfound 2024-04-25T09:22:21.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:315: TEST_rep_backfill_unfound: grep -q obj250 2024-04-25T09:22:22.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:318: TEST_rep_backfill_unfound: timeout 5 rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:27.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:319: TEST_rep_backfill_unfound: test 124 = 124 2024-04-25T09:22:27.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:321: TEST_rep_backfill_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-25T09:22:27.457 INFO:tasks.workunit.client.0.smithi134.stderr:pg has no unfound objects 2024-04-25T09:22:27.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:323: TEST_rep_backfill_unfound: wait_for_clean 2024-04-25T09:22:27.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:22:27.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:22:27.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:22:27.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:22:27.484 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:22:27.484 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:22:27.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:22:27.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:22:27.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:22:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:22:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:22:27.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:22:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:22:27.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:22:27.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:22:28.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:22:28.106 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:22:28.106 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:22:28.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:22:28.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:28.106 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:22:28.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953481 2024-04-25T09:22:28.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953481 2024-04-25T09:22:28.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953481' 2024-04-25T09:22:28.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:28.275 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:22:28.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607594 2024-04-25T09:22:28.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607594 2024-04-25T09:22:28.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953481 1-51539607594' 2024-04-25T09:22:28.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:22:28.523 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:22:28.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345957 2024-04-25T09:22:28.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345957 2024-04-25T09:22:28.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953481 1-51539607594 2-85899345957' 2024-04-25T09:22:28.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:28.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-137438953481 2024-04-25T09:22:28.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:28.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:22:28.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-137438953481 2024-04-25T09:22:28.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:28.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953481 2024-04-25T09:22:28.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 137438953481' 2024-04-25T09:22:28.679 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 137438953481 2024-04-25T09:22:28.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:22:29.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953478 -lt 137438953481 2024-04-25T09:22:29.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:22:30.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:22:30.099 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:22:30.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953481 -lt 137438953481 2024-04-25T09:22:30.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:30.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-51539607594 2024-04-25T09:22:30.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:30.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:22:30.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-51539607594 2024-04-25T09:22:30.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:30.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607594 2024-04-25T09:22:30.622 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 51539607594 2024-04-25T09:22:30.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 51539607594' 2024-04-25T09:22:30.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:22:31.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607594 -lt 51539607594 2024-04-25T09:22:31.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:22:31.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:22:31.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345957 2024-04-25T09:22:31.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:22:31.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345957 2024-04-25T09:22:31.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:22:31.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345957 2024-04-25T09:22:31.238 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 85899345957 2024-04-25T09:22:31.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345957' 2024-04-25T09:22:31.238 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:22:31.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345958 -lt 85899345957 2024-04-25T09:22:31.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:22:31.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:31.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:32.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:22:32.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:32.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:32.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:32.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:32.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:32.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:32.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:32.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:22:32.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:32.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:32.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:33.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:22:33.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' -1 2024-04-25T09:22:33.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:22:33.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=8 2024-04-25T09:22:33.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:22:33.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:22:33.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:22:33.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:33.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:33.363 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:33.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:33.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:33.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:33.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:33.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:22:33.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:33.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:33.786 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:34.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:22:34.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:22:34.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:22:34.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:22:34.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:22:34.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:22:34.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:22:34.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:22:34.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:22:34.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12799 2024-04-25T09:22:34.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12799 '!=' null 2024-04-25T09:22:34.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:22:34.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:22:34.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:22:34.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:22:34.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:34.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:34.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:34.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:34.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:34.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:34.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:35.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:22:35.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:35.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:35.347 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:35.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:22:35.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:22:35.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:22:35.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:22:35.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:22:35.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:22:35.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:22:35.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:22:35.876 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:22:36.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=12799 2024-04-25T09:22:36.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 12799 '!=' null 2024-04-25T09:22:36.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:22:36.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:22:36.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:22:36.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:22:36.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:36.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:36.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:22:36.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:36.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:36.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:37.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 9 2024-04-25T09:22:37.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 8 '!=' 8 2024-04-25T09:22:37.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:22:37.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:22:37.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:22:37.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:22:37.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:22:37.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:22:37.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:22:38.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:22:38.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:22:38.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T09:22:38.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:22:38.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T09:22:38.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:22:38.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:22:38.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:22:38.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:22:38.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:22:38.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:22:38.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:22:38.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:22:38.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:22:38.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:22:38.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:22:38.649 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:22:39.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:22:39.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:22:39.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:22:39.182 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: seq 1 300 2024-04-25T09:22:39.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj1 = obj250 ']' 2024-04-25T09:22:39.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj2 = obj250 ']' 2024-04-25T09:22:39.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj3 = obj250 ']' 2024-04-25T09:22:39.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj4 = obj250 ']' 2024-04-25T09:22:39.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj5 = obj250 ']' 2024-04-25T09:22:39.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj6 = obj250 ']' 2024-04-25T09:22:39.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj7 = obj250 ']' 2024-04-25T09:22:39.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj8 = obj250 ']' 2024-04-25T09:22:39.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj9 = obj250 ']' 2024-04-25T09:22:39.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj10 = obj250 ']' 2024-04-25T09:22:39.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj11 = obj250 ']' 2024-04-25T09:22:39.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj12 = obj250 ']' 2024-04-25T09:22:39.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj13 = obj250 ']' 2024-04-25T09:22:39.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj14 = obj250 ']' 2024-04-25T09:22:39.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj15 = obj250 ']' 2024-04-25T09:22:39.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj16 = obj250 ']' 2024-04-25T09:22:39.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj17 = obj250 ']' 2024-04-25T09:22:39.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj18 = obj250 ']' 2024-04-25T09:22:39.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj19 = obj250 ']' 2024-04-25T09:22:39.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj20 = obj250 ']' 2024-04-25T09:22:39.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj21 = obj250 ']' 2024-04-25T09:22:39.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj22 = obj250 ']' 2024-04-25T09:22:39.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:39.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:39.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj23 = obj250 ']' 2024-04-25T09:22:39.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj24 = obj250 ']' 2024-04-25T09:22:40.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj25 = obj250 ']' 2024-04-25T09:22:40.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj26 = obj250 ']' 2024-04-25T09:22:40.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj27 = obj250 ']' 2024-04-25T09:22:40.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj28 = obj250 ']' 2024-04-25T09:22:40.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj29 = obj250 ']' 2024-04-25T09:22:40.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj30 = obj250 ']' 2024-04-25T09:22:40.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj31 = obj250 ']' 2024-04-25T09:22:40.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj32 = obj250 ']' 2024-04-25T09:22:40.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj33 = obj250 ']' 2024-04-25T09:22:40.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj34 = obj250 ']' 2024-04-25T09:22:40.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj35 = obj250 ']' 2024-04-25T09:22:40.351 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj36 = obj250 ']' 2024-04-25T09:22:40.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj37 = obj250 ']' 2024-04-25T09:22:40.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj38 = obj250 ']' 2024-04-25T09:22:40.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj39 = obj250 ']' 2024-04-25T09:22:40.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj40 = obj250 ']' 2024-04-25T09:22:40.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj41 = obj250 ']' 2024-04-25T09:22:40.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj42 = obj250 ']' 2024-04-25T09:22:40.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj43 = obj250 ']' 2024-04-25T09:22:40.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj44 = obj250 ']' 2024-04-25T09:22:40.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj45 = obj250 ']' 2024-04-25T09:22:40.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj46 = obj250 ']' 2024-04-25T09:22:40.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj47 = obj250 ']' 2024-04-25T09:22:40.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj48 = obj250 ']' 2024-04-25T09:22:40.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj49 = obj250 ']' 2024-04-25T09:22:40.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj50 = obj250 ']' 2024-04-25T09:22:40.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj51 = obj250 ']' 2024-04-25T09:22:40.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj52 = obj250 ']' 2024-04-25T09:22:40.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj53 = obj250 ']' 2024-04-25T09:22:40.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj54 = obj250 ']' 2024-04-25T09:22:40.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj55 = obj250 ']' 2024-04-25T09:22:40.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:40.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:40.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj56 = obj250 ']' 2024-04-25T09:22:40.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj57 = obj250 ']' 2024-04-25T09:22:41.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj58 = obj250 ']' 2024-04-25T09:22:41.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj59 = obj250 ']' 2024-04-25T09:22:41.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj60 = obj250 ']' 2024-04-25T09:22:41.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj61 = obj250 ']' 2024-04-25T09:22:41.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj62 = obj250 ']' 2024-04-25T09:22:41.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj63 = obj250 ']' 2024-04-25T09:22:41.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj64 = obj250 ']' 2024-04-25T09:22:41.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj65 = obj250 ']' 2024-04-25T09:22:41.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj66 = obj250 ']' 2024-04-25T09:22:41.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj67 = obj250 ']' 2024-04-25T09:22:41.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj68 = obj250 ']' 2024-04-25T09:22:41.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj69 = obj250 ']' 2024-04-25T09:22:41.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj70 = obj250 ']' 2024-04-25T09:22:41.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj71 = obj250 ']' 2024-04-25T09:22:41.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj72 = obj250 ']' 2024-04-25T09:22:41.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj73 = obj250 ']' 2024-04-25T09:22:41.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj74 = obj250 ']' 2024-04-25T09:22:41.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj75 = obj250 ']' 2024-04-25T09:22:41.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj76 = obj250 ']' 2024-04-25T09:22:41.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj77 = obj250 ']' 2024-04-25T09:22:41.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj78 = obj250 ']' 2024-04-25T09:22:41.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj79 = obj250 ']' 2024-04-25T09:22:41.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:41.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:41.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj80 = obj250 ']' 2024-04-25T09:22:41.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj81 = obj250 ']' 2024-04-25T09:22:42.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj82 = obj250 ']' 2024-04-25T09:22:42.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj83 = obj250 ']' 2024-04-25T09:22:42.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj84 = obj250 ']' 2024-04-25T09:22:42.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj85 = obj250 ']' 2024-04-25T09:22:42.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj86 = obj250 ']' 2024-04-25T09:22:42.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj87 = obj250 ']' 2024-04-25T09:22:42.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj88 = obj250 ']' 2024-04-25T09:22:42.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj89 = obj250 ']' 2024-04-25T09:22:42.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj90 = obj250 ']' 2024-04-25T09:22:42.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj91 = obj250 ']' 2024-04-25T09:22:42.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj92 = obj250 ']' 2024-04-25T09:22:42.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.444 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj93 = obj250 ']' 2024-04-25T09:22:42.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj94 = obj250 ']' 2024-04-25T09:22:42.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj95 = obj250 ']' 2024-04-25T09:22:42.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj96 = obj250 ']' 2024-04-25T09:22:42.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj97 = obj250 ']' 2024-04-25T09:22:42.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj98 = obj250 ']' 2024-04-25T09:22:42.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj99 = obj250 ']' 2024-04-25T09:22:42.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj100 = obj250 ']' 2024-04-25T09:22:42.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj101 = obj250 ']' 2024-04-25T09:22:42.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj101 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj102 = obj250 ']' 2024-04-25T09:22:42.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj102 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj103 = obj250 ']' 2024-04-25T09:22:42.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj103 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj104 = obj250 ']' 2024-04-25T09:22:42.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj104 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj105 = obj250 ']' 2024-04-25T09:22:42.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj105 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj106 = obj250 ']' 2024-04-25T09:22:42.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj106 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj107 = obj250 ']' 2024-04-25T09:22:42.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj107 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj108 = obj250 ']' 2024-04-25T09:22:42.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj108 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj109 = obj250 ']' 2024-04-25T09:22:42.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj109 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj110 = obj250 ']' 2024-04-25T09:22:42.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj110 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:42.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:42.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj111 = obj250 ']' 2024-04-25T09:22:42.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj111 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj112 = obj250 ']' 2024-04-25T09:22:43.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj112 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.039 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj113 = obj250 ']' 2024-04-25T09:22:43.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj113 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj114 = obj250 ']' 2024-04-25T09:22:43.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj114 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj115 = obj250 ']' 2024-04-25T09:22:43.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj115 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj116 = obj250 ']' 2024-04-25T09:22:43.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj116 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj117 = obj250 ']' 2024-04-25T09:22:43.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj117 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj118 = obj250 ']' 2024-04-25T09:22:43.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj118 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj119 = obj250 ']' 2024-04-25T09:22:43.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj119 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj120 = obj250 ']' 2024-04-25T09:22:43.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj120 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj121 = obj250 ']' 2024-04-25T09:22:43.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj121 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj122 = obj250 ']' 2024-04-25T09:22:43.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj122 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj123 = obj250 ']' 2024-04-25T09:22:43.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj123 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj124 = obj250 ']' 2024-04-25T09:22:43.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj124 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj125 = obj250 ']' 2024-04-25T09:22:43.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj125 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj126 = obj250 ']' 2024-04-25T09:22:43.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj126 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj127 = obj250 ']' 2024-04-25T09:22:43.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj127 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj128 = obj250 ']' 2024-04-25T09:22:43.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj128 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj129 = obj250 ']' 2024-04-25T09:22:43.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj129 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.564 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj130 = obj250 ']' 2024-04-25T09:22:43.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj130 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj131 = obj250 ']' 2024-04-25T09:22:43.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj131 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj132 = obj250 ']' 2024-04-25T09:22:43.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj132 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj133 = obj250 ']' 2024-04-25T09:22:43.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj133 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj134 = obj250 ']' 2024-04-25T09:22:43.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj134 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj135 = obj250 ']' 2024-04-25T09:22:43.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj135 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj136 = obj250 ']' 2024-04-25T09:22:43.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj136 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj137 = obj250 ']' 2024-04-25T09:22:43.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj137 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj138 = obj250 ']' 2024-04-25T09:22:43.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj138 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj139 = obj250 ']' 2024-04-25T09:22:43.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj139 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj140 = obj250 ']' 2024-04-25T09:22:43.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj140 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj141 = obj250 ']' 2024-04-25T09:22:43.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj141 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:43.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:43.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj142 = obj250 ']' 2024-04-25T09:22:43.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj142 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj143 = obj250 ']' 2024-04-25T09:22:44.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj143 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj144 = obj250 ']' 2024-04-25T09:22:44.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj144 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj145 = obj250 ']' 2024-04-25T09:22:44.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj145 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj146 = obj250 ']' 2024-04-25T09:22:44.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj146 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj147 = obj250 ']' 2024-04-25T09:22:44.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj147 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj148 = obj250 ']' 2024-04-25T09:22:44.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj148 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj149 = obj250 ']' 2024-04-25T09:22:44.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj149 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj150 = obj250 ']' 2024-04-25T09:22:44.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj150 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj151 = obj250 ']' 2024-04-25T09:22:44.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj151 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj152 = obj250 ']' 2024-04-25T09:22:44.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj152 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj153 = obj250 ']' 2024-04-25T09:22:44.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj153 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj154 = obj250 ']' 2024-04-25T09:22:44.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj154 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.360 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj155 = obj250 ']' 2024-04-25T09:22:44.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj155 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj156 = obj250 ']' 2024-04-25T09:22:44.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj156 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj157 = obj250 ']' 2024-04-25T09:22:44.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj157 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj158 = obj250 ']' 2024-04-25T09:22:44.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj158 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj159 = obj250 ']' 2024-04-25T09:22:44.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj159 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj160 = obj250 ']' 2024-04-25T09:22:44.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj160 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj161 = obj250 ']' 2024-04-25T09:22:44.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj161 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj162 = obj250 ']' 2024-04-25T09:22:44.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj162 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj163 = obj250 ']' 2024-04-25T09:22:44.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj163 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj164 = obj250 ']' 2024-04-25T09:22:44.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj164 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj165 = obj250 ']' 2024-04-25T09:22:44.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj165 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj166 = obj250 ']' 2024-04-25T09:22:44.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj166 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj167 = obj250 ']' 2024-04-25T09:22:44.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj167 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj168 = obj250 ']' 2024-04-25T09:22:44.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj168 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj169 = obj250 ']' 2024-04-25T09:22:44.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj169 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj170 = obj250 ']' 2024-04-25T09:22:44.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj170 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj171 = obj250 ']' 2024-04-25T09:22:44.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj171 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj172 = obj250 ']' 2024-04-25T09:22:44.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj172 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj173 = obj250 ']' 2024-04-25T09:22:44.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj173 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj174 = obj250 ']' 2024-04-25T09:22:44.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj174 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj175 = obj250 ']' 2024-04-25T09:22:44.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj175 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:44.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:44.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj176 = obj250 ']' 2024-04-25T09:22:44.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj176 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj177 = obj250 ']' 2024-04-25T09:22:45.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj177 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj178 = obj250 ']' 2024-04-25T09:22:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj178 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj179 = obj250 ']' 2024-04-25T09:22:45.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj179 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj180 = obj250 ']' 2024-04-25T09:22:45.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj180 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj181 = obj250 ']' 2024-04-25T09:22:45.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj181 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj182 = obj250 ']' 2024-04-25T09:22:45.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj182 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj183 = obj250 ']' 2024-04-25T09:22:45.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj183 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj184 = obj250 ']' 2024-04-25T09:22:45.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj184 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj185 = obj250 ']' 2024-04-25T09:22:45.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj185 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj186 = obj250 ']' 2024-04-25T09:22:45.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj186 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj187 = obj250 ']' 2024-04-25T09:22:45.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj187 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj188 = obj250 ']' 2024-04-25T09:22:45.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj188 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj189 = obj250 ']' 2024-04-25T09:22:45.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj189 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj190 = obj250 ']' 2024-04-25T09:22:45.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj190 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj191 = obj250 ']' 2024-04-25T09:22:45.455 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj191 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj192 = obj250 ']' 2024-04-25T09:22:45.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj192 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj193 = obj250 ']' 2024-04-25T09:22:45.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj193 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj194 = obj250 ']' 2024-04-25T09:22:45.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj194 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj195 = obj250 ']' 2024-04-25T09:22:45.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj195 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj196 = obj250 ']' 2024-04-25T09:22:45.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj196 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj197 = obj250 ']' 2024-04-25T09:22:45.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj197 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj198 = obj250 ']' 2024-04-25T09:22:45.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj198 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj199 = obj250 ']' 2024-04-25T09:22:45.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj199 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj200 = obj250 ']' 2024-04-25T09:22:45.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj200 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj201 = obj250 ']' 2024-04-25T09:22:45.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj201 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj202 = obj250 ']' 2024-04-25T09:22:45.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj202 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj203 = obj250 ']' 2024-04-25T09:22:45.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj203 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj204 = obj250 ']' 2024-04-25T09:22:45.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj204 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj205 = obj250 ']' 2024-04-25T09:22:45.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj205 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj206 = obj250 ']' 2024-04-25T09:22:45.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj206 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:45.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:45.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj207 = obj250 ']' 2024-04-25T09:22:45.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj207 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj208 = obj250 ']' 2024-04-25T09:22:46.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj208 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj209 = obj250 ']' 2024-04-25T09:22:46.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj209 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj210 = obj250 ']' 2024-04-25T09:22:46.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj210 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj211 = obj250 ']' 2024-04-25T09:22:46.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj211 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj212 = obj250 ']' 2024-04-25T09:22:46.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj212 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj213 = obj250 ']' 2024-04-25T09:22:46.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj213 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj214 = obj250 ']' 2024-04-25T09:22:46.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj214 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj215 = obj250 ']' 2024-04-25T09:22:46.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj215 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj216 = obj250 ']' 2024-04-25T09:22:46.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj216 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj217 = obj250 ']' 2024-04-25T09:22:46.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj217 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj218 = obj250 ']' 2024-04-25T09:22:46.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj218 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj219 = obj250 ']' 2024-04-25T09:22:46.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj219 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj220 = obj250 ']' 2024-04-25T09:22:46.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj220 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj221 = obj250 ']' 2024-04-25T09:22:46.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj221 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj222 = obj250 ']' 2024-04-25T09:22:46.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj222 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj223 = obj250 ']' 2024-04-25T09:22:46.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj223 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj224 = obj250 ']' 2024-04-25T09:22:46.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj224 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj225 = obj250 ']' 2024-04-25T09:22:46.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj225 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj226 = obj250 ']' 2024-04-25T09:22:46.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj226 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj227 = obj250 ']' 2024-04-25T09:22:46.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj227 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj228 = obj250 ']' 2024-04-25T09:22:46.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj228 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj229 = obj250 ']' 2024-04-25T09:22:46.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj229 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj230 = obj250 ']' 2024-04-25T09:22:46.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj230 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj231 = obj250 ']' 2024-04-25T09:22:46.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj231 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj232 = obj250 ']' 2024-04-25T09:22:46.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj232 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj233 = obj250 ']' 2024-04-25T09:22:46.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj233 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj234 = obj250 ']' 2024-04-25T09:22:46.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj234 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj235 = obj250 ']' 2024-04-25T09:22:46.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj235 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj236 = obj250 ']' 2024-04-25T09:22:46.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj236 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj237 = obj250 ']' 2024-04-25T09:22:46.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj237 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj238 = obj250 ']' 2024-04-25T09:22:46.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj238 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj239 = obj250 ']' 2024-04-25T09:22:46.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj239 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj240 = obj250 ']' 2024-04-25T09:22:46.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj240 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:46.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:46.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj241 = obj250 ']' 2024-04-25T09:22:46.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj241 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj242 = obj250 ']' 2024-04-25T09:22:47.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj242 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj243 = obj250 ']' 2024-04-25T09:22:47.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj243 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj244 = obj250 ']' 2024-04-25T09:22:47.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj244 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj245 = obj250 ']' 2024-04-25T09:22:47.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj245 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj246 = obj250 ']' 2024-04-25T09:22:47.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj246 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj247 = obj250 ']' 2024-04-25T09:22:47.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj247 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj248 = obj250 ']' 2024-04-25T09:22:47.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj248 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj249 = obj250 ']' 2024-04-25T09:22:47.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj249 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj250 = obj250 ']' 2024-04-25T09:22:47.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:329: TEST_rep_backfill_unfound: rados -p test-pool get obj250 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.280 INFO:tasks.workunit.client.0.smithi134.stderr:error getting test-pool/obj250: (2) No such file or directory 2024-04-25T09:22:47.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj251 = obj250 ']' 2024-04-25T09:22:47.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj251 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj252 = obj250 ']' 2024-04-25T09:22:47.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj252 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj253 = obj250 ']' 2024-04-25T09:22:47.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj253 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj254 = obj250 ']' 2024-04-25T09:22:47.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj254 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj255 = obj250 ']' 2024-04-25T09:22:47.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj255 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj256 = obj250 ']' 2024-04-25T09:22:47.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj256 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj257 = obj250 ']' 2024-04-25T09:22:47.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj257 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj258 = obj250 ']' 2024-04-25T09:22:47.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj258 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj259 = obj250 ']' 2024-04-25T09:22:47.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj259 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj260 = obj250 ']' 2024-04-25T09:22:47.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj260 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj261 = obj250 ']' 2024-04-25T09:22:47.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj261 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.625 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj262 = obj250 ']' 2024-04-25T09:22:47.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj262 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj263 = obj250 ']' 2024-04-25T09:22:47.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj263 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj264 = obj250 ']' 2024-04-25T09:22:47.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj264 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj265 = obj250 ']' 2024-04-25T09:22:47.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj265 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj266 = obj250 ']' 2024-04-25T09:22:47.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj266 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj267 = obj250 ']' 2024-04-25T09:22:47.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj267 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj268 = obj250 ']' 2024-04-25T09:22:47.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj268 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj269 = obj250 ']' 2024-04-25T09:22:47.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj269 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj270 = obj250 ']' 2024-04-25T09:22:47.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj270 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj271 = obj250 ']' 2024-04-25T09:22:47.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj271 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:47.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:47.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj272 = obj250 ']' 2024-04-25T09:22:47.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj272 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj273 = obj250 ']' 2024-04-25T09:22:48.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj273 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj274 = obj250 ']' 2024-04-25T09:22:48.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj274 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj275 = obj250 ']' 2024-04-25T09:22:48.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj275 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj276 = obj250 ']' 2024-04-25T09:22:48.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj276 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj277 = obj250 ']' 2024-04-25T09:22:48.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj277 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj278 = obj250 ']' 2024-04-25T09:22:48.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj278 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj279 = obj250 ']' 2024-04-25T09:22:48.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj279 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj280 = obj250 ']' 2024-04-25T09:22:48.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj280 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj281 = obj250 ']' 2024-04-25T09:22:48.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj281 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj282 = obj250 ']' 2024-04-25T09:22:48.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj282 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj283 = obj250 ']' 2024-04-25T09:22:48.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj283 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj284 = obj250 ']' 2024-04-25T09:22:48.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj284 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj285 = obj250 ']' 2024-04-25T09:22:48.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj285 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj286 = obj250 ']' 2024-04-25T09:22:48.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj286 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj287 = obj250 ']' 2024-04-25T09:22:48.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj287 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj288 = obj250 ']' 2024-04-25T09:22:48.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj288 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj289 = obj250 ']' 2024-04-25T09:22:48.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj289 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj290 = obj250 ']' 2024-04-25T09:22:48.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj290 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj291 = obj250 ']' 2024-04-25T09:22:48.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj291 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj292 = obj250 ']' 2024-04-25T09:22:48.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj292 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj293 = obj250 ']' 2024-04-25T09:22:48.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj293 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj294 = obj250 ']' 2024-04-25T09:22:48.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj294 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj295 = obj250 ']' 2024-04-25T09:22:48.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj295 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj296 = obj250 ']' 2024-04-25T09:22:48.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj296 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj297 = obj250 ']' 2024-04-25T09:22:48.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj297 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.747 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj298 = obj250 ']' 2024-04-25T09:22:48.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj298 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj299 = obj250 ']' 2024-04-25T09:22:48.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj299 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:325: TEST_rep_backfill_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:22:48.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:327: TEST_rep_backfill_unfound: '[' obj300 = obj250 ']' 2024-04-25T09:22:48.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:331: TEST_rep_backfill_unfound: rados --pool test-pool get obj300 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:332: TEST_rep_backfill_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:336: TEST_rep_backfill_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:22:48.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:338: TEST_rep_backfill_unfound: delete_pool test-pool 2024-04-25T09:22:48.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-04-25T09:22:48.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-04-25T09:22:49.341 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test-pool' does not exist 2024-04-25T09:22:49.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:22:49.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:22:49.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:22:49.359 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:22:49.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:22:49.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:22:49.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:22:49.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:22:49.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:22:49.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:22:49.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:22:49.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:22:49.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:22:49.675 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:22:49.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:22:49.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:22:49.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:22:49.676 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:22:49.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:22:49.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:22:49.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:22:49.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:49.724 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:22:49.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.390086 2024-04-25T09:22:49.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:22:49.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:22:49.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:35: run: for func in $funcs 2024-04-25T09:22:49.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:36: run: setup td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:22:49.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:22:49.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:22:49.729 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:22:49.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:22:49.730 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:22:49.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:22:49.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:22:49.731 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:22:49.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:22:49.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:22:49.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:22:49.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:22:49.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:22:49.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:22:49.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.736 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:22:49.736 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:49.736 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:22:49.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.390086 2024-04-25T09:22:49.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:22:49.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:22:49.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:22:49.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:49.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:22:49.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.390086 2024-04-25T09:22:49.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-rep-recov-eio.sh 1' TERM HUP INT 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:38: run: run_mon td/osd-rep-recov-eio.sh a --mon_osd_warn_num_repaired=10 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-rep-recov-eio.sh/a 2024-04-25T09:22:49.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-rep-recov-eio.sh/a --run-dir=td/osd-rep-recov-eio.sh --mon_osd_warn_num_repaired=10 2024-04-25T09:22:50.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:22:50.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:22:50.235 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:22:50.235 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:22:50.235 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:50.235 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:22:50.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:22:50.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-rep-recov-eio.sh/a '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --mon-cluster-log-file=td/osd-rep-recov-eio.sh/log --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_osd_warn_num_repaired=10 2024-04-25T09:22:50.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:22:50.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:22:50.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:22:50.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:22:50.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:22:50.465 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:22:50.465 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:22:50.466 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:22:50.466 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:22:50.467 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:22:50.467 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:50.467 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:22:50.467 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-mon.a.asok 2024-04-25T09:22:50.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:22:50.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.390086/ceph-mon.a.asok config get fsid 2024-04-25T09:22:50.578 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:22:50.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:22:50.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:22:50.579 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:22:50.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:22:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:22:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:22:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:22:50.580 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:22:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:22:50.581 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-mon.a.asok 2024-04-25T09:22:50.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:22:50.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.390086/ceph-mon.a.asok config get mon_host 2024-04-25T09:22:50.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:39: run: run_mgr td/osd-rep-recov-eio.sh x 2024-04-25T09:22:50.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:22:50.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:22:50.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:22:50.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:22:50.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-rep-recov-eio.sh/x 2024-04-25T09:22:50.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:22:51.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:22:51.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:22:51.629 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:22:51.630 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:22:51.630 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:51.630 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:22:51.630 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:22:51.631 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:22:51.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-rep-recov-eio.sh/x '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --run-dir=td/osd-rep-recov-eio.sh '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:22:51.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:40: run: ceph osd pool create foo 8 2024-04-25T09:22:52.050 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T09:22:52.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:42: run: TEST_rep_recovery_unfound td/osd-rep-recov-eio.sh 2024-04-25T09:22:52.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:343: TEST_rep_recovery_unfound: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:22:52.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:344: TEST_rep_recovery_unfound: local objname=myobject 2024-04-25T09:22:52.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:345: TEST_rep_recovery_unfound: local lastobj=100 2024-04-25T09:22:52.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:347: TEST_rep_recovery_unfound: local testobj=obj75 2024-04-25T09:22:52.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:349: TEST_rep_recovery_unfound: setup_osds 3 2024-04-25T09:22:52.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:48: setup_osds: local count=3 2024-04-25T09:22:52.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:49: setup_osds: shift 2024-04-25T09:22:52.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:50: setup_osds: local type= 2024-04-25T09:22:52.069 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: expr 3 - 1 2024-04-25T09:22:52.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: seq 0 2 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:22:52.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:22:52.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:22:52.076 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:22:52.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8d08dba1-0ca5-4fed-ac17-1f44379fa759 2024-04-25T09:22:52.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8d08dba1-0ca5-4fed-ac17-1f44379fa759' 2024-04-25T09:22:52.077 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 8d08dba1-0ca5-4fed-ac17-1f44379fa759 2024-04-25T09:22:52.077 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:22:52.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsICpmDZHnBhAArwiZfFt2ol4jkp06pFIuPw== 2024-04-25T09:22:52.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsICpmDZHnBhAArwiZfFt2ol4jkp06pFIuPw=="}' 2024-04-25T09:22:52.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8d08dba1-0ca5-4fed-ac17-1f44379fa759 -i td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:22:52.443 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:22:52.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/0/new.json 2024-04-25T09:22:52.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsICpmDZHnBhAArwiZfFt2ol4jkp06pFIuPw== --osd-uuid 8d08dba1-0ca5-4fed-ac17-1f44379fa759 2024-04-25T09:22:52.497 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:52.507+0000 7fdbcd5cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:52.519 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:52.531+0000 7fdbcd5cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:52.520 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:52.531+0000 7fdbcd5cc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:52.521 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:52.531+0000 7fdbcd5cc7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:22:52.521 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:52.531+0000 7fdbcd5cc7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:22:52.521 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:52.531+0000 7fdbcd5cc7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/0/block: (2) No such file or directory 2024-04-25T09:22:52.559 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:52.571+0000 7fdbcd5cc7c0 -1 bluestore(td/osd-rep-recov-eio.sh/0) _read_fsid unparsable uuid 2024-04-25T09:22:59.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/0/keyring 2024-04-25T09:22:59.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:22:59.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:22:59.057 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:22:59.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:22:59.626 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:22:59.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:22:59.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:22:59.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:22:59.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:22:59.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:22:59.661 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:59.671+0000 7fd60f2977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:59.709 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:59.719+0000 7fd60f2977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:22:59.711 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:22:59.723+0000 7fd60f2977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:00.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:23:00.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:23:00.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:23:00.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:23:00.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:23:00.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:00.043 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:23:00.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:23:00.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:00.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:00.284 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:00.295+0000 7fd60f2977c0 -1 Falling back to public interface 2024-04-25T09:23:00.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:01.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:01.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:01.466 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:23:01.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:23:01.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:01.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:01.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:02.881 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:02.891+0000 7fd60f2977c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:23:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:02.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:02.897 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:23:02.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:23:02.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:02.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:03.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:04.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:04.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:04.571 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:23:04.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:23:04.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:04.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:05.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:06.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:06.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:06.131 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:23:06.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:23:06.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:06.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:06.690 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:07.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:07.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:07.692 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:23:07.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:23:07.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:07.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:08.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:09.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:09.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:09.140 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:23:09.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:23:09.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:09.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:09.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:10.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:10.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:10.733 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:23:10.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:23:10.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:10.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:11.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:12.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:12.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:12.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:23:12.195 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:23:12.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:12.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:12.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:13.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:13.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:13.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:23:13.790 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:23:13.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:13.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:14.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:15.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:15.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:15.224 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:23:15.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:23:15.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:15.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:15.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:16.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:16.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:16.722 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:23:16.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:23:16.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:16.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:17.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:18.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:18.725 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:23:18.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:23:18.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:18.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:19.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:20.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:20.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:20.681 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:23:20.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:23:20.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:20.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:21.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:22.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:22.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:22.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:23:22.853 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:23:22.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:22.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:23.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:24.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:24.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:24.736 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:23:24.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:23:24.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:24.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:25.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:26.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:26.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:26.626 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:23:26.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:23:26.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:26.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:27.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:28.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:28.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:28.475 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:23:28.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:23:28.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:28.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:23:29.106 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 6 up_thru 6 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/11153152,v1:127.0.0.1:6803/11153152] [v2:127.0.0.1:6804/11153152,v1:127.0.0.1:6805/11153152] exists,up 8d08dba1-0ca5-4fed-ac17-1f44379fa759 2024-04-25T09:23:29.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:23:29.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:23:29.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:23:29.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:23:29.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 1 2024-04-25T09:23:29.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:23:29.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/1 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/1' 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/1/journal' 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:23:29.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:23:29.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:23:29.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:23:29.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:23:29.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/1 2024-04-25T09:23:29.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:23:29.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d65c7c7f-ce79-4f31-a3d9-f997e6ccb6bb 2024-04-25T09:23:29.115 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 d65c7c7f-ce79-4f31-a3d9-f997e6ccb6bb 2024-04-25T09:23:29.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d65c7c7f-ce79-4f31-a3d9-f997e6ccb6bb' 2024-04-25T09:23:29.115 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:23:29.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARISpmagDKCBAA8s3SOe0UueV339vXa8imkA== 2024-04-25T09:23:29.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARISpmagDKCBAA8s3SOe0UueV339vXa8imkA=="}' 2024-04-25T09:23:29.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d65c7c7f-ce79-4f31-a3d9-f997e6ccb6bb -i td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:23:29.785 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:23:29.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/1/new.json 2024-04-25T09:23:29.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARISpmagDKCBAA8s3SOe0UueV339vXa8imkA== --osd-uuid d65c7c7f-ce79-4f31-a3d9-f997e6ccb6bb 2024-04-25T09:23:29.837 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:29.846+0000 7eff4846c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:29.876 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:29.886+0000 7eff4846c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:29.877 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:29.890+0000 7eff4846c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:29.878 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:29.890+0000 7eff4846c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:23:29.878 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:29.890+0000 7eff4846c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:23:29.878 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:29.890+0000 7eff4846c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/1/block: (2) No such file or directory 2024-04-25T09:23:29.916 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:29.926+0000 7eff4846c7c0 -1 bluestore(td/osd-rep-recov-eio.sh/1) _read_fsid unparsable uuid 2024-04-25T09:23:36.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/1/keyring 2024-04-25T09:23:36.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:23:36.402 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:23:36.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:23:36.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:23:37.004 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:23:37.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:23:37.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/1 --osd-journal=td/osd-rep-recov-eio.sh/1/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:23:37.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:23:37.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:23:37.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:23:37.033 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:37.042+0000 7f26d919e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:37.065 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:37.074+0000 7f26d919e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:37.066 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:37.078+0000 7f26d919e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:23:37.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:23:37.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:23:37.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:23:37.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:23:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:23:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:23:37.436 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:23:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:37.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:37.603 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:37.614+0000 7f26d919e7c0 -1 Falling back to public interface 2024-04-25T09:23:37.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:38.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:38.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:38.873 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:23:38.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:23:38.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:38.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:39.303 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:40.266 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:23:40.278+0000 7f26d919e7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:23:40.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:40.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:40.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:23:40.305 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:23:40.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:40.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:40.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:41.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:41.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:41.907 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:23:41.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:23:41.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:41.908 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:42.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:43.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:43.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:43.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:23:43.346 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:23:43.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:43.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:43.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:44.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:44.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:44.881 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:23:44.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:23:44.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:44.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:45.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:46.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:46.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:46.453 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:23:46.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:23:46.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:46.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:46.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:47.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:47.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:47.888 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:23:47.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:23:47.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:47.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:48.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:49.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:49.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:49.417 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:23:49.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:23:49.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:49.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:49.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:50.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:50.846 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:23:50.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:50.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:23:50.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:50.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:51.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:52.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:52.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:52.580 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:23:52.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:23:52.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:52.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:53.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:54.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:54.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:54.129 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:23:54.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:23:54.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:54.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:54.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:55.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:55.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:55.598 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:23:55.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:23:55.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:55.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:56.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:57.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:57.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:57.759 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:23:57.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:23:57.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:57.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:23:58.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:23:59.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:23:59.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:23:59.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:23:59.611 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:23:59.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:23:59.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:24:00.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:01.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:01.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:01.519 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:24:01.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:24:01.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:01.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:24:02.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:03.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:03.282 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:03.283 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:24:03.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:24:03.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:03.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:24:03.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:04.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:04.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:04.846 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:24:04.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:24:04.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:04.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:24:05.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:06.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:06.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:06.621 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:24:06.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:24:06.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:06.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:24:07.238 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1591275571,v1:127.0.0.1:6811/1591275571] [v2:127.0.0.1:6812/1591275571,v1:127.0.0.1:6813/1591275571] exists,up d65c7c7f-ce79-4f31-a3d9-f997e6ccb6bb 2024-04-25T09:24:07.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:24:07.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:52: setup_osds: for id in $(seq 0 $(expr $count - 1)) 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:53: setup_osds: run_osd td/osd-rep-recov-eio.sh 2 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-rep-recov-eio.sh/2 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:24:07.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/2' 2024-04-25T09:24:07.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/2/journal' 2024-04-25T09:24:07.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:24:07.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:24:07.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:24:07.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:24:07.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:24:07.241 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:24:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:24:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:24:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:24:07.243 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:24:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:24:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:24:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:24:07.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-rep-recov-eio.sh/2 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=67563c87-3802-44cf-a023-e7911edeb962 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 67563c87-3802-44cf-a023-e7911edeb962' 2024-04-25T09:24:07.244 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 67563c87-3802-44cf-a023-e7911edeb962 2024-04-25T09:24:07.245 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:24:07.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3ISpmWxbPEBAAFGEYjJwVJahd6kTB73TNvg== 2024-04-25T09:24:07.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3ISpmWxbPEBAAFGEYjJwVJahd6kTB73TNvg=="}' 2024-04-25T09:24:07.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 67563c87-3802-44cf-a023-e7911edeb962 -i td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:24:08.128 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:24:08.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-rep-recov-eio.sh/2/new.json 2024-04-25T09:24:08.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3ISpmWxbPEBAAFGEYjJwVJahd6kTB73TNvg== --osd-uuid 67563c87-3802-44cf-a023-e7911edeb962 2024-04-25T09:24:08.187 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:08.201+0000 7f5babf877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:08.256 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:08.269+0000 7f5babf877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:08.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:08.273+0000 7f5babf877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:08.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:08.273+0000 7f5babf877c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:24:08.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:08.273+0000 7f5babf877c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:24:08.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:08.273+0000 7f5babf877c0 -1 bluestore(td/osd-rep-recov-eio.sh/2/block) _read_bdev_label failed to open td/osd-rep-recov-eio.sh/2/block: (2) No such file or directory 2024-04-25T09:24:08.399 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:08.413+0000 7f5babf877c0 -1 bluestore(td/osd-rep-recov-eio.sh/2) _read_fsid unparsable uuid 2024-04-25T09:24:15.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-rep-recov-eio.sh/2/keyring 2024-04-25T09:24:15.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:24:15.392 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:24:15.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:24:15.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-rep-recov-eio.sh/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:24:15.945 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:24:15.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:24:15.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:24:15.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:24:15.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:24:15.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/2 --osd-journal=td/osd-rep-recov-eio.sh/2/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:24:15.980 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:15.993+0000 7f98df92d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:16.014 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:16.029+0000 7f98df92d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:16.015 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:16.029+0000 7f98df92d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:24:16.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:24:16.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:24:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:24:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:24:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:24:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:16.358 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:24:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:24:16.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:16.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:16.583 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:16.597+0000 7f98df92d7c0 -1 Falling back to public interface 2024-04-25T09:24:16.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:17.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:17.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:17.811 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:24:17.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:24:17.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:17.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:18.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:19.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:19.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:19.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:24:19.229 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:24:19.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:19.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:19.332 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:24:19.345+0000 7f98df92d7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:24:19.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:20.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:20.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:20.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:24:20.717 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:24:20.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:20.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:21.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:22.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:22.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:22.425 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:24:22.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:24:22.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:22.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:22.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:23.877 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:24:23.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:24:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:23.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:24.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:25.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:25.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:25.378 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:24:25.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:24:25.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:25.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:25.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:26.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:26.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:26.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:24:26.955 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:24:26.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:26.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:27.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:28.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:28.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:28.549 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:24:28.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:24:28.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:28.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:29.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:30.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:30.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:30.114 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:24:30.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:24:30.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:30.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:30.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:31.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:31.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:31.752 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:24:31.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:24:31.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:31.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:32.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:33.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:33.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:33.338 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:24:33.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:24:33.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:33.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:33.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:34.851 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:34.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:34.852 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:24:34.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:24:34.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:34.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:35.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:36.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:36.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:36.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:24:36.642 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:24:36.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:36.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:37.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:38.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:38.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:38.746 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:24:38.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:24:38.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:38.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:40.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:41.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:41.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:41.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:24:41.051 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:24:41.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:41.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:41.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:42.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:42.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:42.498 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:24:42.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:24:42.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:42.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:43.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:44.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:44.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:44.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:24:44.243 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:24:44.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:44.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:44.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:24:45.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:24:45.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:24:45.750 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:24:45.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:24:45.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:24:45.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:24:46.299 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/457969686,v1:127.0.0.1:6819/457969686] [v2:127.0.0.1:6820/457969686,v1:127.0.0.1:6821/457969686] exists,up 67563c87-3802-44cf-a023-e7911edeb962 2024-04-25T09:24:46.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:24:46.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:24:46.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:24:46.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:55: setup_osds: wait_for_clean 2024-04-25T09:24:46.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:24:46.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:24:46.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:24:46.301 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:24:46.302 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:24:46.302 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:24:46.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:24:46.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:24:46.302 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:24:46.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:24:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:24:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:24:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:24:46.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:24:46.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:24:46.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:24:46.910 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:24:46.910 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:24:46.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:24:46.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:46.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:24:47.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803797 2024-04-25T09:24:47.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803797 2024-04-25T09:24:47.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803797' 2024-04-25T09:24:47.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:47.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:24:47.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574863 2024-04-25T09:24:47.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574863 2024-04-25T09:24:47.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803797 1-55834574863' 2024-04-25T09:24:47.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:47.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:24:47.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345928 2024-04-25T09:24:47.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345928 2024-04-25T09:24:47.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803797 1-55834574863 2-85899345928' 2024-04-25T09:24:47.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:47.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803797 2024-04-25T09:24:47.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:47.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:24:47.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803797 2024-04-25T09:24:47.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:47.384 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803797 2024-04-25T09:24:47.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803797 2024-04-25T09:24:47.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803797' 2024-04-25T09:24:47.385 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:47.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803795 -lt 25769803797 2024-04-25T09:24:47.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:24:48.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:24:48.996 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:49.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803797 -lt 25769803797 2024-04-25T09:24:49.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:49.451 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574863 2024-04-25T09:24:49.451 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:49.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:24:49.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574863 2024-04-25T09:24:49.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:49.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574863 2024-04-25T09:24:49.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574863' 2024-04-25T09:24:49.453 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574863 2024-04-25T09:24:49.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:24:49.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574864 -lt 55834574863 2024-04-25T09:24:49.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:49.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:49.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345928 2024-04-25T09:24:49.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:24:49.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345928 2024-04-25T09:24:49.957 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:49.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345928 2024-04-25T09:24:49.959 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 85899345928 2024-04-25T09:24:49.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345928' 2024-04-25T09:24:49.959 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:24:50.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345928 -lt 85899345928 2024-04-25T09:24:50.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:24:50.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:50.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:50.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-04-25T09:24:50.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:24:50.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:24:50.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:24:50.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:24:50.947 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:24:50.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:24:50.948 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:24:51.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:24:51.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:24:51.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:51.338 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:51.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 8 2024-04-25T09:24:51.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:24:51.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:24:51.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:24:51.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:24:51.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:24:51.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:24:51.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:24:51.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:24:51.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:24:51.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:24:51.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:24:51.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:24:51.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:24:52.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:24:52.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:24:52.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:52.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:52.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 8 2024-04-25T09:24:52.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 0 2024-04-25T09:24:52.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:24:52.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=2 2024-04-25T09:24:52.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:24:52.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:24:53.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:24:53.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:24:53.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:24:53.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:24:53.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:24:53.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:24:53.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:24:53.020 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:24:53.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:24:53.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:24:53.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:53.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:53.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 8 2024-04-25T09:24:53.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 2 '!=' 2 2024-04-25T09:24:53.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:24:53.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:24:53.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:24:53.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:24:53.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:24:53.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:24:53.984 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:24:54.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-04-25T09:24:54.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-04-25T09:24:54.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-04-25T09:24:54.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:24:54.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-04-25T09:24:54.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:24:54.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:24:54.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:24:54.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:24:54.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:24:54.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:24:54.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:24:54.771 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:24:55.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-04-25T09:24:55.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:24:55.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:24:55.179 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:24:55.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-04-25T09:24:55.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:24:55.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:24:55.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:351: TEST_rep_recovery_unfound: local poolname=test-pool 2024-04-25T09:24:55.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:352: TEST_rep_recovery_unfound: create_pool test-pool 1 1 2024-04-25T09:24:55.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test-pool 1 1 2024-04-25T09:24:56.262 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test-pool' already exists 2024-04-25T09:24:56.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:24:57.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:353: TEST_rep_recovery_unfound: wait_for_clean 2024-04-25T09:24:57.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:24:57.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:24:57.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:24:57.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:24:57.282 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:24:57.282 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:24:57.282 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:24:57.283 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:24:57.283 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:24:57.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:24:57.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:24:57.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:24:57.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:24:57.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:24:57.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:24:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:24:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:24:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:24:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:24:57.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:57.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:24:58.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=25769803801 2024-04-25T09:24:58.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 25769803801 2024-04-25T09:24:58.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801' 2024-04-25T09:24:58.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:58.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:24:58.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574868 2024-04-25T09:24:58.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574868 2024-04-25T09:24:58.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801 1-55834574868' 2024-04-25T09:24:58.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:24:58.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:24:58.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345932 2024-04-25T09:24:58.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345932 2024-04-25T09:24:58.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-25769803801 1-55834574868 2-85899345932' 2024-04-25T09:24:58.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:24:58.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-25769803801 2024-04-25T09:24:58.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:24:58.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:24:58.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-25769803801 2024-04-25T09:24:58.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:24:58.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=25769803801 2024-04-25T09:24:58.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 25769803801' 2024-04-25T09:24:58.583 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 25769803801 2024-04-25T09:24:58.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:24:59.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803799 -lt 25769803801 2024-04-25T09:24:59.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:25:00.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:25:00.035 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:25:00.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 25769803802 -lt 25769803801 2024-04-25T09:25:00.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:25:00.475 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574868 2024-04-25T09:25:00.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:25:00.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:25:00.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574868 2024-04-25T09:25:00.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:25:00.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574868 2024-04-25T09:25:00.479 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574868 2024-04-25T09:25:00.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574868' 2024-04-25T09:25:00.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:25:00.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574868 -lt 55834574868 2024-04-25T09:25:00.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:25:00.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345932 2024-04-25T09:25:00.915 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:25:00.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:25:00.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345932 2024-04-25T09:25:00.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:25:00.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345932 2024-04-25T09:25:00.918 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 85899345932 2024-04-25T09:25:00.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345932' 2024-04-25T09:25:00.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:25:01.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345932 -lt 85899345932 2024-04-25T09:25:01.356 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:25:01.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:25:01.357 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:25:01.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:25:01.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:25:01.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:25:01.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:25:01.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:25:01.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:25:01.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:25:01.932 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:25:02.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:25:02.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:25:02.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:25:02.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:25:02.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:25:02.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:25:02.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:25:02.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:355: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:25:03.360 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:25:03.360 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.360198+0000 0'0 25:32 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:16:03.391666+0000 0 0 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.361034+0000 0'0 25:32 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:52:46.112394+0000 0 0 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:49.449916+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:44:29.470606+0000 0 0 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.360277+0000 0'0 25:32 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:45:34.617901+0000 0 0 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:57.973925+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:24:56.102245+0000 0'0 2024-04-25T09:24:56.102245+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:59:37.966624+0000 0 0 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.360295+0000 0'0 25:32 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:21:57.438854+0000 0 0 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:49.449578+0000 0'0 25:47 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:04:42.054015+0000 0 0 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:49.340797+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:29:49.201297+0000 0 0 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.360258+0000 0'0 25:32 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:59:05.540493+0000 0 0 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T09:25:03.361 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:25:03.362 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T09:25:03.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:357: TEST_rep_recovery_unfound: rados_put td/osd-rep-recov-eio.sh test-pool myobject 2024-04-25T09:25:03.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:66: rados_put: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:25:03.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:67: rados_put: local poolname=test-pool 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:68: rados_put: local objname=myobject 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 AAA 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 BBB 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 CCCC 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:70: rados_put: for marker in AAA BBB CCCC DDDD 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:71: rados_put: printf '%*s' 1024 DDDD 2024-04-25T09:25:03.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:76: rados_put: rados --pool test-pool put myobject td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:03.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: get_osds test-pool myobject 2024-04-25T09:25:03.535 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:25:03.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=myobject 2024-04-25T09:25:03.536 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool myobject 2024-04-25T09:25:03.536 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:25:04.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:25:04.054 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:25:04.054 INFO:tasks.workunit.client.0.smithi134.stderr:0' 2024-04-25T09:25:04.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 0 2024-04-25T09:25:04.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: initial_osds=('2' '1' '0') 2024-04-25T09:25:04.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:359: TEST_rep_recovery_unfound: local -a initial_osds 2024-04-25T09:25:04.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:360: TEST_rep_recovery_unfound: local last_osd=0 2024-04-25T09:25:04.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:361: TEST_rep_recovery_unfound: kill_daemons td/osd-rep-recov-eio.sh TERM osd.0 2024-04-25T09:25:04.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:25:04.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:25:04.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:25:04.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:25:04.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:25:04.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:25:04.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:362: TEST_rep_recovery_unfound: ceph osd down 0 2024-04-25T09:25:05.098 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already down. 2024-04-25T09:25:05.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:363: TEST_rep_recovery_unfound: ceph osd out 0 2024-04-25T09:25:05.738 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. 2024-04-25T09:25:05.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:365: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:25:06.187 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:25:06.187 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.360198+0000 0'0 26:34 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:16:03.391666+0000 0 0 2024-04-25T09:25:06.187 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.361034+0000 0'0 26:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T19:52:46.112394+0000 0 0 2024-04-25T09:25:06.187 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:49.449916+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T09:44:29.470606+0000 0 0 2024-04-25T09:25:06.187 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.360277+0000 0'0 26:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:45:34.617901+0000 0 0 2024-04-25T09:25:06.187 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:57.973925+0000 0'0 26:13 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:24:56.102245+0000 0'0 2024-04-25T09:24:56.102245+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:59:37.966624+0000 0 0 2024-04-25T09:25:06.187 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.360295+0000 0'0 26:34 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T14:21:57.438854+0000 0 0 2024-04-25T09:25:06.188 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 stale+active+clean 2024-04-25T09:24:49.449578+0000 0'0 26:49 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:04:42.054015+0000 0 0 2024-04-25T09:25:06.188 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:49.340797+0000 0'0 26:18 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:29:49.201297+0000 0 0 2024-04-25T09:25:06.188 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:24:48.360258+0000 0'0 26:34 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:59:05.540493+0000 0 0 2024-04-25T09:25:06.188 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T09:25:06.188 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:25:06.188 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T09:25:06.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:367: TEST_rep_recovery_unfound: dd if=/dev/urandom of=td/osd-rep-recov-eio.sh/ORIGINAL bs=1024 count=4 2024-04-25T09:25:06.205 INFO:tasks.workunit.client.0.smithi134.stderr:4+0 records in 2024-04-25T09:25:06.205 INFO:tasks.workunit.client.0.smithi134.stderr:4+0 records out 2024-04-25T09:25:06.205 INFO:tasks.workunit.client.0.smithi134.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000177354 s, 23.1 MB/s 2024-04-25T09:25:06.206 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:25:06.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:06.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj1 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:06.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:06.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj2 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:06.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:06.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj3 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:06.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:06.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj4 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:06.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:06.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj5 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:07.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:07.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj6 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:07.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:07.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj7 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:07.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:07.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj8 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:07.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:07.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj9 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:07.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:07.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj10 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:07.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:07.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj11 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:08.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:08.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj12 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:08.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:08.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj13 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:08.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:08.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj14 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:08.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:08.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj15 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:08.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:08.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj16 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:08.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:08.873 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj17 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:09.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:09.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj18 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:09.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:09.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj19 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:09.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:09.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj20 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:09.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:09.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj21 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:09.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:09.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj22 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:09.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:09.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj23 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:09.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:09.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj24 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:10.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:10.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj25 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:10.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:10.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj26 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:10.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:10.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj27 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:10.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:10.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj28 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:10.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:10.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj29 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:11.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:11.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj30 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:11.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:11.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj31 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:11.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:11.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj32 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:11.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:11.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj33 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:11.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:11.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj34 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:11.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:11.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj35 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:12.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:12.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj36 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:12.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:12.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj37 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:12.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:12.361 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj38 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:12.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:12.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj39 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:12.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:12.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj40 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:12.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:12.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj41 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:13.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:13.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj42 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:13.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:13.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj43 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:13.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:13.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj44 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:13.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:13.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj45 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:13.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:13.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj46 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:13.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:13.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj47 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:14.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:14.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj48 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:14.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:14.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj49 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:14.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:14.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj50 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:14.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:14.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj51 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:14.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:14.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj52 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:14.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:14.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj53 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:15.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:15.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj54 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:15.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:15.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj55 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:15.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:15.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj56 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:15.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:15.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj57 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:15.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:15.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj58 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:15.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:15.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj59 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:16.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:16.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj60 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:16.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:16.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj61 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:16.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:16.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj62 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:16.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:16.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj63 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:16.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:16.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj64 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:16.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:16.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj65 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:17.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:17.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj66 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:17.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:17.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj67 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:17.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:17.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj68 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:17.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:17.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj69 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:17.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:17.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj70 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:17.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:17.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj71 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:18.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:18.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj72 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:18.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:18.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj73 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:18.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:18.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj74 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:18.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:18.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj75 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:18.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:18.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj76 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:18.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:18.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj77 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:18.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:18.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj78 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:19.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:19.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj79 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:19.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:19.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj80 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:19.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:19.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj81 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:19.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:19.624 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj82 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:19.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:19.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj83 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:19.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:19.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj84 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:20.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:20.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj85 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:20.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:20.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj86 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:20.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:20.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj87 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:20.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:20.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj88 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:20.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:20.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj89 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:20.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:20.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj90 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:21.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:21.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj91 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:21.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:21.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj92 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:21.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj93 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:21.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:21.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj94 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:21.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:21.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj95 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:21.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:21.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj96 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:22.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:22.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj97 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:22.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:22.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj98 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:22.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:22.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj99 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:22.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:368: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:22.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:370: TEST_rep_recovery_unfound: rados --pool test-pool put obj100 td/osd-rep-recov-eio.sh/ORIGINAL 2024-04-25T09:25:22.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:373: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 0 2024-04-25T09:25:22.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:25:22.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:25:22.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:25:22.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:25:22.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:25:22.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:25:22.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj75 2024-04-25T09:25:22.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:25:22.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:25:22.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:25:22.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=0 2024-04-25T09:25:22.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:25:22.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj75 2024-04-25T09:25:22.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:25:22.715 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-25T09:25:22.715 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-04-25T09:25:22.715 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:25:23.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:25:23.157 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T09:25:23.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:25:23.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:25:23.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:25:23.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=2 2024-04-25T09:25:23.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:25:23.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:25:23.158 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/2/type 2024-04-25T09:25:23.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:25:23.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 2 bluestore_debug_inject_read_err true 2024-04-25T09:25:23.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:25:23.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=2 2024-04-25T09:25:23.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:25:23.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:25:23.160 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:25:23.160 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.2 2024-04-25T09:25:23.160 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:25:23.160 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:25:23.160 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:25:23.160 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:23.160 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:25:23.160 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.2.asok 2024-04-25T09:25:23.161 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.2.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:25:23.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:25:23.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:25:23.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:25:23.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.2 2024-04-25T09:25:23.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-04-25T09:25:23.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-04-25T09:25:23.281 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:25:23.281 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:23.281 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:25:23.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.2.asok 2024-04-25T09:25:23.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:25:23.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.2.asok injectdataerr test-pool obj75 2024-04-25T09:25:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:374: TEST_rep_recovery_unfound: inject_eio rep data test-pool obj75 td/osd-rep-recov-eio.sh 1 2024-04-25T09:25:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2410: inject_eio: local pooltype=rep 2024-04-25T09:25:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2411: inject_eio: shift 2024-04-25T09:25:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2412: inject_eio: local which=data 2024-04-25T09:25:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2413: inject_eio: shift 2024-04-25T09:25:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: inject_eio: local poolname=test-pool 2024-04-25T09:25:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2415: inject_eio: shift 2024-04-25T09:25:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: inject_eio: local objname=obj75 2024-04-25T09:25:23.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: inject_eio: shift 2024-04-25T09:25:23.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: inject_eio: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:25:23.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: inject_eio: shift 2024-04-25T09:25:23.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: inject_eio: local shard_id=1 2024-04-25T09:25:23.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2421: inject_eio: shift 2024-04-25T09:25:23.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: get_osds test-pool obj75 2024-04-25T09:25:23.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=test-pool 2024-04-25T09:25:23.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=obj75 2024-04-25T09:25:23.402 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map test-pool obj75 2024-04-25T09:25:23.402 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-04-25T09:25:23.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=2 2024-04-25T09:25:23.810 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T09:25:23.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 2 1 2024-04-25T09:25:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: initial_osds=('2' '1') 2024-04-25T09:25:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: inject_eio: local -a initial_osds 2024-04-25T09:25:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2424: inject_eio: local osd_id=1 2024-04-25T09:25:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: inject_eio: '[' rep '!=' ec ']' 2024-04-25T09:25:23.811 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: inject_eio: shard_id= 2024-04-25T09:25:23.811 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: cat td/osd-rep-recov-eio.sh/1/type 2024-04-25T09:25:23.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2428: inject_eio: type=bluestore 2024-04-25T09:25:23.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2429: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2024-04-25T09:25:23.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=osd 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=1 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=bluestore_debug_inject_read_err 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path osd.1 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:23.815 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:25:23.816 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:25:23.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.390086/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2024-04-25T09:25:23.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-04-25T09:25:23.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2430: inject_eio: local loop=0 2024-04-25T09:25:23.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2432: inject_eio: grep -q Invalid 2024-04-25T09:25:23.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: get_asok_path osd.1 2024-04-25T09:25:23.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-04-25T09:25:23.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-04-25T09:25:23.940 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:25:23.940 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:23.940 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:25:23.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.390086/ceph-osd.1.asok 2024-04-25T09:25:23.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: CEPH_ARGS= 2024-04-25T09:25:23.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2431: inject_eio: ceph --admin-daemon /tmp/ceph-asok.390086/ceph-osd.1.asok injectdataerr test-pool obj75 2024-04-25T09:25:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:376: TEST_rep_recovery_unfound: activate_osd td/osd-rep-recov-eio.sh 0 2024-04-25T09:25:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:25:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:25:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:25:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:25:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/osd-rep-recov-eio.sh/0 2024-04-25T09:25:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10' 2024-04-25T09:25:24.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/osd-rep-recov-eio.sh/0' 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/osd-rep-recov-eio.sh/0/journal' 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/osd-rep-recov-eio.sh' 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:25:24.050 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/osd-rep-recov-eio.sh/$name.log' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/osd-rep-recov-eio.sh/$name.pid' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:25:24.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/osd-rep-recov-eio.sh/0 2024-04-25T09:25:24.053 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:25:24.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:25:24.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=52ea6452-1094-49d7-af80-9f2c0397539e --auth-supported=none --mon-host=127.0.0.1:7140 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-rep-recov-eio.sh/0 --osd-journal=td/osd-rep-recov-eio.sh/0/journal --chdir= --run-dir=td/osd-rep-recov-eio.sh '--admin-socket=/tmp/ceph-asok.390086/$cluster-$name.asok' --debug-osd=20 '--log-file=td/osd-rep-recov-eio.sh/$name.log' '--pid-file=td/osd-rep-recov-eio.sh/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:25:24.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/osd-rep-recov-eio.sh/0/whoami 2024-04-25T09:25:24.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:25:24.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:25:24.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:25:24.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:25:24.079 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:25:24.092+0000 7f276f65e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:24.082 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:25:24.096+0000 7f276f65e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:24.083 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:25:24.096+0000 7f276f65e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:25:24.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:25:24.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:25:24.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:25:24.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:25:24.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:25:24.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:24.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:25:24.545 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:25:24.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:24.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:24.628 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:25:24.640+0000 7f276f65e7c0 -1 Falling back to public interface 2024-04-25T09:25:24.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:25.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:25.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:25.947 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:25:25.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:25:25.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:25.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:26.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:27.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:27.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:27.376 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:25:27.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:25:27.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:27.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:27.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:25:27.968 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:25:27.980+0000 7f276f65e7c0 -1 osd.0 26 log_to_monitors true 2024-04-25T09:25:28.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:25:28.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:25:28.898 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:25:28.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:25:28.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:25:28.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:25:29.730 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up out weight 0 up_from 32 up_thru 21 down_at 27 last_clean_interval [6,26) [v2:127.0.0.1:6802/3162170377,v1:127.0.0.1:6803/3162170377] [v2:127.0.0.1:6804/3162170377,v1:127.0.0.1:6805/3162170377] exists,up 8d08dba1-0ca5-4fed-ac17-1f44379fa759 2024-04-25T09:25:29.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:25:29.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:25:29.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:25:29.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:377: TEST_rep_recovery_unfound: ceph osd in 0 2024-04-25T09:25:30.953 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already in. 2024-04-25T09:25:30.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:379: TEST_rep_recovery_unfound: sleep 15 2024-04-25T09:25:45.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:25:45.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:381: TEST_rep_recovery_unfound: for tmp in $(seq 1 100) 2024-04-25T09:25:45.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: get_state 2.0 2024-04-25T09:25:45.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:59: get_state: local pgid=2.0 2024-04-25T09:25:45.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:60: get_state: local sname=state 2024-04-25T09:25:45.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:61: get_state: ceph --format json pg dump pgs 2024-04-25T09:25:45.975 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:62: get_state: jq -r '.pg_stats | .[] | select(.pgid=="2.0") | .state' 2024-04-25T09:25:46.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:382: TEST_rep_recovery_unfound: state=active+recovery_unfound+degraded 2024-04-25T09:25:46.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: echo active+recovery_unfound+degraded 2024-04-25T09:25:46.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:383: TEST_rep_recovery_unfound: grep -v recovering 2024-04-25T09:25:46.408 INFO:tasks.workunit.client.0.smithi134.stdout:active+recovery_unfound+degraded 2024-04-25T09:25:46.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:384: TEST_rep_recovery_unfound: '[' 0 = 0 ']' 2024-04-25T09:25:46.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:385: TEST_rep_recovery_unfound: break 2024-04-25T09:25:46.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:391: TEST_rep_recovery_unfound: ceph pg dump pgs 2024-04-25T09:25:46.796 INFO:tasks.workunit.client.0.smithi134.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN LAST_SCRUB_DURATION SCRUB_SCHEDULING OBJECTS_SCRUBBED OBJECTS_TRIMMED 2024-04-25T09:25:46.796 INFO:tasks.workunit.client.0.smithi134.stdout:1.7 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:25:31.962305+0000 0'0 40:67 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T17:34:53.252897+0000 0 0 2024-04-25T09:25:46.796 INFO:tasks.workunit.client.0.smithi134.stdout:1.6 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:25:32.951733+0000 0'0 40:66 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T15:06:57.902643+0000 0 0 2024-04-25T09:25:46.797 INFO:tasks.workunit.client.0.smithi134.stdout:1.5 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:25:33.084370+0000 0'0 40:51 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T16:57:36.511136+0000 0 0 2024-04-25T09:25:46.797 INFO:tasks.workunit.client.0.smithi134.stdout:1.4 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:25:32.951226+0000 0'0 40:66 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T11:53:40.589300+0000 0 0 2024-04-25T09:25:46.797 INFO:tasks.workunit.client.0.smithi134.stdout:2.0 101 1 3 0 1 413696 0 0 101 0 101 active+recovery_unfound+degraded 2024-04-25T09:25:37.698235+0000 30'101 40:349 [2,1,0] 2 [2,1,0] 2 0'0 2024-04-25T09:24:56.102245+0000 0'0 2024-04-25T09:24:56.102245+0000 0 0 no scrub is scheduled 0 0 2024-04-25T09:25:46.797 INFO:tasks.workunit.client.0.smithi134.stdout:1.3 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:25:31.886787+0000 0'0 40:67 [1,2,0] 1 [1,2,0] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T20:29:44.674170+0000 0 0 2024-04-25T09:25:46.797 INFO:tasks.workunit.client.0.smithi134.stdout:1.2 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:25:32.766099+0000 0'0 40:57 [0,1,2] 0 [0,1,2] 0 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:47:20.091673+0000 0 0 2024-04-25T09:25:46.797 INFO:tasks.workunit.client.0.smithi134.stdout:1.1 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:25:33.250258+0000 0'0 40:57 [2,0,1] 2 [2,0,1] 2 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T18:48:28.536132+0000 0 0 2024-04-25T09:25:46.797 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 0 0 0 0 0 0 0 0 0 0 0 active+clean 2024-04-25T09:25:32.952373+0000 0'0 40:66 [1,0,2] 1 [1,0,2] 1 0'0 2024-04-25T09:22:51.913934+0000 0'0 2024-04-25T09:22:51.913934+0000 0 0 periodic scrub scheduled @ 2024-04-26T12:15:40.499622+0000 0 0 2024-04-25T09:25:46.797 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T09:25:46.797 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:25:46.798 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pgs 2024-04-25T09:25:46.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: ceph pg 2.0 list_unfound 2024-04-25T09:25:46.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:392: TEST_rep_recovery_unfound: grep -q obj75 2024-04-25T09:25:46.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:395: TEST_rep_recovery_unfound: timeout 5 rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:51.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:396: TEST_rep_recovery_unfound: test 124 = 124 2024-04-25T09:25:51.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:398: TEST_rep_recovery_unfound: ceph pg 2.0 mark_unfound_lost delete 2024-04-25T09:25:52.211 INFO:tasks.workunit.client.0.smithi134.stderr:pg has no unfound objects 2024-04-25T09:25:52.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:400: TEST_rep_recovery_unfound: wait_for_clean 2024-04-25T09:25:52.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:25:52.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:25:52.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:25:52.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:25:52.228 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:25:52.228 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:25:52.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:25:52.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:25:52.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:25:52.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:25:52.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:25:52.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:25:52.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:25:52.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:25:52.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:25:52.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:25:52.778 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:25:52.778 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:25:52.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:25:52.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:25:52.778 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:25:52.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=137438953479 2024-04-25T09:25:52.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 137438953479 2024-04-25T09:25:52.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953479' 2024-04-25T09:25:52.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:25:52.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:25:53.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=55834574881 2024-04-25T09:25:53.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 55834574881 2024-04-25T09:25:53.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953479 1-55834574881' 2024-04-25T09:25:53.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:25:53.095 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:25:53.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345945 2024-04-25T09:25:53.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345945 2024-04-25T09:25:53.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-137438953479 1-55834574881 2-85899345945' 2024-04-25T09:25:53.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:25:53.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-137438953479 2024-04-25T09:25:53.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:25:53.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:25:53.250 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-137438953479 2024-04-25T09:25:53.250 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:25:53.251 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 137438953479 2024-04-25T09:25:53.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=137438953479 2024-04-25T09:25:53.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 137438953479' 2024-04-25T09:25:53.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:25:53.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953477 -lt 137438953479 2024-04-25T09:25:53.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:25:54.719 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:25:54.720 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:25:55.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 137438953479 -lt 137438953479 2024-04-25T09:25:55.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:25:55.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-55834574881 2024-04-25T09:25:55.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:25:55.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:25:55.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-55834574881 2024-04-25T09:25:55.139 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:25:55.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=55834574881 2024-04-25T09:25:55.140 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 55834574881 2024-04-25T09:25:55.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 55834574881' 2024-04-25T09:25:55.140 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:25:55.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 55834574881 -lt 55834574881 2024-04-25T09:25:55.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:25:55.558 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-85899345945 2024-04-25T09:25:55.558 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:25:55.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:25:55.559 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-85899345945 2024-04-25T09:25:55.559 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:25:55.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345945 2024-04-25T09:25:55.561 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 85899345945' 2024-04-25T09:25:55.561 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 85899345945 2024-04-25T09:25:55.562 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:25:56.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345945 -lt 85899345945 2024-04-25T09:25:56.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:25:56.010 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:25:56.010 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:25:56.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 9 == 0 2024-04-25T09:25:56.695 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:25:56.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:25:56.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:25:56.696 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:25:56.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:25:56.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:25:56.697 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:25:57.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=9 2024-04-25T09:25:57.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:25:57.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:25:57.075 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:25:57.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 9 = 9 2024-04-25T09:25:57.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:25:57.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:25:57.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: seq 1 100 2024-04-25T09:25:57.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:57.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj1 = obj75 ']' 2024-04-25T09:25:57.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj1 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:57.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj2 = obj75 ']' 2024-04-25T09:25:57.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj2 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj3 = obj75 ']' 2024-04-25T09:25:57.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj3 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:57.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj4 = obj75 ']' 2024-04-25T09:25:57.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj4 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:57.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj5 = obj75 ']' 2024-04-25T09:25:57.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj5 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:57.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj6 = obj75 ']' 2024-04-25T09:25:57.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj6 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:57.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj7 = obj75 ']' 2024-04-25T09:25:57.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj7 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:57.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj8 = obj75 ']' 2024-04-25T09:25:57.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj8 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:57.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:57.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj9 = obj75 ']' 2024-04-25T09:25:57.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj9 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj10 = obj75 ']' 2024-04-25T09:25:58.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj10 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj11 = obj75 ']' 2024-04-25T09:25:58.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj11 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj12 = obj75 ']' 2024-04-25T09:25:58.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj12 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj13 = obj75 ']' 2024-04-25T09:25:58.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj13 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj14 = obj75 ']' 2024-04-25T09:25:58.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj14 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj15 = obj75 ']' 2024-04-25T09:25:58.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj15 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj16 = obj75 ']' 2024-04-25T09:25:58.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj16 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj17 = obj75 ']' 2024-04-25T09:25:58.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj17 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj18 = obj75 ']' 2024-04-25T09:25:58.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj18 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj19 = obj75 ']' 2024-04-25T09:25:58.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj19 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj20 = obj75 ']' 2024-04-25T09:25:58.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj20 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj21 = obj75 ']' 2024-04-25T09:25:58.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj21 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj22 = obj75 ']' 2024-04-25T09:25:58.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj22 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj23 = obj75 ']' 2024-04-25T09:25:58.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj23 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj24 = obj75 ']' 2024-04-25T09:25:58.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj24 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj25 = obj75 ']' 2024-04-25T09:25:58.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj25 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj26 = obj75 ']' 2024-04-25T09:25:58.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj26 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj27 = obj75 ']' 2024-04-25T09:25:58.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj27 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj28 = obj75 ']' 2024-04-25T09:25:58.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj28 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj29 = obj75 ']' 2024-04-25T09:25:58.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj29 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj30 = obj75 ']' 2024-04-25T09:25:58.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj30 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj31 = obj75 ']' 2024-04-25T09:25:58.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj31 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj32 = obj75 ']' 2024-04-25T09:25:58.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj32 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj33 = obj75 ']' 2024-04-25T09:25:58.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj33 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj34 = obj75 ']' 2024-04-25T09:25:58.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj34 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj35 = obj75 ']' 2024-04-25T09:25:58.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj35 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj36 = obj75 ']' 2024-04-25T09:25:58.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj36 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj37 = obj75 ']' 2024-04-25T09:25:58.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj37 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj38 = obj75 ']' 2024-04-25T09:25:58.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj38 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj39 = obj75 ']' 2024-04-25T09:25:58.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj39 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj40 = obj75 ']' 2024-04-25T09:25:58.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj40 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:58.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:58.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj41 = obj75 ']' 2024-04-25T09:25:58.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj41 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj42 = obj75 ']' 2024-04-25T09:25:59.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj42 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj43 = obj75 ']' 2024-04-25T09:25:59.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj43 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj44 = obj75 ']' 2024-04-25T09:25:59.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj44 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj45 = obj75 ']' 2024-04-25T09:25:59.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj45 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj46 = obj75 ']' 2024-04-25T09:25:59.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj46 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj47 = obj75 ']' 2024-04-25T09:25:59.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj47 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj48 = obj75 ']' 2024-04-25T09:25:59.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj48 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj49 = obj75 ']' 2024-04-25T09:25:59.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj49 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj50 = obj75 ']' 2024-04-25T09:25:59.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj50 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj51 = obj75 ']' 2024-04-25T09:25:59.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj51 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj52 = obj75 ']' 2024-04-25T09:25:59.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj52 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.346 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj53 = obj75 ']' 2024-04-25T09:25:59.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj53 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj54 = obj75 ']' 2024-04-25T09:25:59.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj54 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.406 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj55 = obj75 ']' 2024-04-25T09:25:59.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj55 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj56 = obj75 ']' 2024-04-25T09:25:59.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj56 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj57 = obj75 ']' 2024-04-25T09:25:59.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj57 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.495 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj58 = obj75 ']' 2024-04-25T09:25:59.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj58 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj59 = obj75 ']' 2024-04-25T09:25:59.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj59 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj60 = obj75 ']' 2024-04-25T09:25:59.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj60 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj61 = obj75 ']' 2024-04-25T09:25:59.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj61 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj62 = obj75 ']' 2024-04-25T09:25:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj62 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj63 = obj75 ']' 2024-04-25T09:25:59.645 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj63 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj64 = obj75 ']' 2024-04-25T09:25:59.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj64 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj65 = obj75 ']' 2024-04-25T09:25:59.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj65 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj66 = obj75 ']' 2024-04-25T09:25:59.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj66 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj67 = obj75 ']' 2024-04-25T09:25:59.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj67 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj68 = obj75 ']' 2024-04-25T09:25:59.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj68 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj69 = obj75 ']' 2024-04-25T09:25:59.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj69 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj70 = obj75 ']' 2024-04-25T09:25:59.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj70 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj71 = obj75 ']' 2024-04-25T09:25:59.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj71 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj72 = obj75 ']' 2024-04-25T09:25:59.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj72 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj73 = obj75 ']' 2024-04-25T09:25:59.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj73 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:25:59.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:25:59.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj74 = obj75 ']' 2024-04-25T09:25:59.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj74 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj75 = obj75 ']' 2024-04-25T09:26:00.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:406: TEST_rep_recovery_unfound: rados -p test-pool get obj75 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.030 INFO:tasks.workunit.client.0.smithi134.stderr:error getting test-pool/obj75: (2) No such file or directory 2024-04-25T09:26:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj76 = obj75 ']' 2024-04-25T09:26:00.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj76 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj77 = obj75 ']' 2024-04-25T09:26:00.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj77 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj78 = obj75 ']' 2024-04-25T09:26:00.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj78 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj79 = obj75 ']' 2024-04-25T09:26:00.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj79 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj80 = obj75 ']' 2024-04-25T09:26:00.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj80 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj81 = obj75 ']' 2024-04-25T09:26:00.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj81 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.207 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj82 = obj75 ']' 2024-04-25T09:26:00.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj82 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj83 = obj75 ']' 2024-04-25T09:26:00.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj83 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj84 = obj75 ']' 2024-04-25T09:26:00.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj84 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj85 = obj75 ']' 2024-04-25T09:26:00.313 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj85 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj86 = obj75 ']' 2024-04-25T09:26:00.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj86 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj87 = obj75 ']' 2024-04-25T09:26:00.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj87 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj88 = obj75 ']' 2024-04-25T09:26:00.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj88 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj89 = obj75 ']' 2024-04-25T09:26:00.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj89 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj90 = obj75 ']' 2024-04-25T09:26:00.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj90 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj91 = obj75 ']' 2024-04-25T09:26:00.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj91 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj92 = obj75 ']' 2024-04-25T09:26:00.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj92 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj93 = obj75 ']' 2024-04-25T09:26:00.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj93 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj94 = obj75 ']' 2024-04-25T09:26:00.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj94 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj95 = obj75 ']' 2024-04-25T09:26:00.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj95 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj96 = obj75 ']' 2024-04-25T09:26:00.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj96 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj97 = obj75 ']' 2024-04-25T09:26:00.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj97 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj98 = obj75 ']' 2024-04-25T09:26:00.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj98 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj99 = obj75 ']' 2024-04-25T09:26:00.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj99 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:402: TEST_rep_recovery_unfound: for i in $(seq 1 $lastobj) 2024-04-25T09:26:00.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:404: TEST_rep_recovery_unfound: '[' obj100 = obj75 ']' 2024-04-25T09:26:00.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:408: TEST_rep_recovery_unfound: rados --pool test-pool get obj100 td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:409: TEST_rep_recovery_unfound: diff -q td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:413: TEST_rep_recovery_unfound: rm -f td/osd-rep-recov-eio.sh/ORIGINAL td/osd-rep-recov-eio.sh/CHECK 2024-04-25T09:26:00.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:415: TEST_rep_recovery_unfound: delete_pool test-pool 2024-04-25T09:26:00.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test-pool 2024-04-25T09:26:00.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test-pool test-pool --yes-i-really-really-mean-it 2024-04-25T09:26:01.305 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test-pool' does not exist 2024-04-25T09:26:01.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-rep-recov-eio.sh:43: run: teardown td/osd-rep-recov-eio.sh 2024-04-25T09:26:01.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:26:01.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:26:01.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:26:01.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:26:01.322 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:26:01.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:26:01.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:26:01.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:26:01.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:26:01.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:26:01.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:26:01.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:26:01.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:26:01.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:26:01.434 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:26:01.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:01.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:26:01.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:26:01.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:01.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:26:01.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:26:01.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:26:01.462 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:26:01.462 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:01.462 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:26:01.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.390086 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-rep-recov-eio.sh 0 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-rep-recov-eio.sh 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-rep-recov-eio.sh KILL 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:26:01.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:26:01.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:26:01.466 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:26:01.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:26:01.467 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:26:01.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:26:01.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:26:01.469 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:26:01.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:01.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:26:01.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:26:01.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:01.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:26:01.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:26:01.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-rep-recov-eio.sh 2024-04-25T09:26:01.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:26:01.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:01.473 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.390086 2024-04-25T09:26:01.473 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.390086 2024-04-25T09:26:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:26:01.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:26:01.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:26:01.475 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:26:01.475 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:26:01.526 INFO:tasks.workunit:Running workunit osd/osd-reuse-id.sh... 2024-04-25T09:26:01.526 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/osd-reuse-id.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh 2024-04-25T09:26:01.579 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:26:01.584 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:26:01.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:26:01.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:26:01.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:26:01.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:26:01.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:26:01.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:26:01.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:26:01.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:26:01.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/osd-reuse-id 2024-04-25T09:26:01.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:20: run: local dir=td/osd-reuse-id 2024-04-25T09:26:01.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:21: run: shift 2024-04-25T09:26:01.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: export CEPH_MON=127.0.0.1:7123 2024-04-25T09:26:01.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:23: run: CEPH_MON=127.0.0.1:7123 2024-04-25T09:26:01.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:24: run: export CEPH_ARGS 2024-04-25T09:26:01.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: uuidgen 2024-04-25T09:26:01.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:25: run: CEPH_ARGS+='--fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none ' 2024-04-25T09:26:01.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7123 ' 2024-04-25T09:26:01.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: set 2024-04-25T09:26:01.586 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:26:01.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:28: run: local funcs=TEST_reuse_id 2024-04-25T09:26:01.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:29: run: for func in $funcs 2024-04-25T09:26:01.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:30: run: setup td/osd-reuse-id 2024-04-25T09:26:01.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-reuse-id 2024-04-25T09:26:01.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-reuse-id 2024-04-25T09:26:01.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:26:01.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:26:01.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:26:01.588 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:26:01.588 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:26:01.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:26:01.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:26:01.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:26:01.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:26:01.589 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:26:01.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:26:01.590 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:26:01.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:26:01.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:26:01.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:26:01.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:01.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:26:01.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:26:01.592 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:26:01.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:26:01.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:26:01.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:26:01.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:26:01.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:01.595 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:26:01.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.446386 2024-04-25T09:26:01.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:26:01.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:26:01.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-reuse-id 2024-04-25T09:26:01.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:26:01.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:01.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:26:01.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.446386 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-reuse-id 1' TERM HUP INT 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:31: run: TEST_reuse_id td/osd-reuse-id 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:37: TEST_reuse_id: local dir=td/osd-reuse-id 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:39: TEST_reuse_id: run_mon td/osd-reuse-id a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-reuse-id 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:26:01.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:26:01.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:26:01.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-reuse-id/a 2024-04-25T09:26:01.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-reuse-id/a --run-dir=td/osd-reuse-id --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:26:02.085 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:26:02.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:26:02.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:26:02.086 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:26:02.086 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:02.086 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:26:02.086 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.446386/$cluster-$name.asok' 2024-04-25T09:26:02.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-reuse-id/a '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' --mon-cluster-log-file=td/osd-reuse-id/log --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:26:02.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:26:02.290 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:26:02.291 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.446386/ceph-mon.a.asok 2024-04-25T09:26:02.292 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:26:02.292 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.446386/ceph-mon.a.asok config get fsid 2024-04-25T09:26:02.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:26:02.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:26:02.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:26:02.402 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:26:02.403 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:26:02.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:26:02.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:26:02.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:26:02.403 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:26:02.403 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:02.403 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:26:02.403 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.446386/ceph-mon.a.asok 2024-04-25T09:26:02.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:26:02.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.446386/ceph-mon.a.asok config get mon_host 2024-04-25T09:26:02.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:40: TEST_reuse_id: run_mgr td/osd-reuse-id x 2024-04-25T09:26:02.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/osd-reuse-id 2024-04-25T09:26:02.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:26:02.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:26:02.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:26:02.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/osd-reuse-id/x 2024-04-25T09:26:02.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:26:03.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:26:03.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:26:03.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:26:03.435 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:26:03.436 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:03.436 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:26:03.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.446386/$cluster-$name.asok' 2024-04-25T09:26:03.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:26:03.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-reuse-id/x '--log-file=td/osd-reuse-id/$name.log' '--admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' --run-dir=td/osd-reuse-id '--pid-file=td/osd-reuse-id/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:26:03.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:41: TEST_reuse_id: run_osd td/osd-reuse-id 0 2024-04-25T09:26:03.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:26:03.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:26:03.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:26:03.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:26:03.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/0 2024-04-25T09:26:03.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/0' 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/0/journal' 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:26:03.470 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.446386/$cluster-$name.asok' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:26:03.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/0 2024-04-25T09:26:03.472 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:26:03.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4c50886f-a7b9-46e7-bd43-8fb12534e956 2024-04-25T09:26:03.472 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4c50886f-a7b9-46e7-bd43-8fb12534e956' 2024-04-25T09:26:03.473 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 4c50886f-a7b9-46e7-bd43-8fb12534e956 2024-04-25T09:26:03.473 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:26:03.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrISpmQo35HRAA7U4PINnEmTewBpzbRfGG6Q== 2024-04-25T09:26:03.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrISpmQo35HRAA7U4PINnEmTewBpzbRfGG6Q=="}' 2024-04-25T09:26:03.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4c50886f-a7b9-46e7-bd43-8fb12534e956 -i td/osd-reuse-id/0/new.json 2024-04-25T09:26:03.791 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:26:03.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/0/new.json 2024-04-25T09:26:03.808 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCrISpmQo35HRAA7U4PINnEmTewBpzbRfGG6Q== --osd-uuid 4c50886f-a7b9-46e7-bd43-8fb12534e956 2024-04-25T09:26:03.845 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:03.855+0000 7fe484a577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:03.861 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:03.871+0000 7fe484a577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:03.862 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:03.875+0000 7fe484a577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:03.862 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:03.875+0000 7fe484a577c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:26:03.863 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:03.875+0000 7fe484a577c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:26:03.863 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:03.875+0000 7fe484a577c0 -1 bluestore(td/osd-reuse-id/0/block) _read_bdev_label failed to open td/osd-reuse-id/0/block: (2) No such file or directory 2024-04-25T09:26:03.900 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:03.911+0000 7fe484a577c0 -1 bluestore(td/osd-reuse-id/0) _read_fsid unparsable uuid 2024-04-25T09:26:10.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/0/keyring 2024-04-25T09:26:10.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:26:10.431 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:26:10.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:26:10.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:26:10.968 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:26:10.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:26:10.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:26:10.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:26:10.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:26:10.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/0 --osd-journal=td/osd-reuse-id/0/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:26:10.998 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:11.011+0000 7efd72ab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:11.000 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:11.011+0000 7efd72ab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:11.001 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:11.015+0000 7efd72ab37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:11.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:26:11.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:26:11.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:26:11.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:26:11.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:26:11.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:11.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:26:11.435 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:26:11.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:11.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:11.562 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:11.575+0000 7efd72ab37c0 -1 Falling back to public interface 2024-04-25T09:26:11.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:12.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:12.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:12.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:26:12.924 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:26:12.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:12.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:13.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:14.157 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:14.171+0000 7efd72ab37c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:26:14.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:14.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:14.391 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:26:14.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:26:14.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:14.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:15.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:16.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:16.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:16.007 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:26:16.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:26:16.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:16.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:16.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:17.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:17.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:17.449 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:26:17.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:26:17.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:17.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:18.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:19.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:19.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:19.059 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:26:19.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:26:19.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:19.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:19.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:20.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:20.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:20.502 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:26:20.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:26:20.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:20.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:21.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:22.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:22.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:22.043 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:26:22.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:26:22.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:22.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:22.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:23.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:23.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:23.466 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:26:23.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:26:23.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:23.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:23.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:24.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:24.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:24.989 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:26:24.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:26:24.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:24.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:25.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:26.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:26.427 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:26:26.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:26:26.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:26.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:27.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:28.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:28.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:28.068 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:26:28.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:26:28.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:28.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:29.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:30.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:30.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:30.373 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:26:30.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:26:30.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:30.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:31.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:32.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:32.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:32.383 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:26:32.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:26:32.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:32.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:32.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:33.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:33.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:33.952 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:26:33.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:26:33.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:33.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:35.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:36.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:36.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:36.308 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:26:36.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:26:36.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:36.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:37.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:38.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:38.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:38.165 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:26:38.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:26:38.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:38.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:39.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:40.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:40.100 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:26:40.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:26:40.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:40.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:26:40.502 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3271148579,v1:127.0.0.1:6803/3271148579] [v2:127.0.0.1:6804/3271148579,v1:127.0.0.1:6805/3271148579] exists,up 4c50886f-a7b9-46e7-bd43-8fb12534e956 2024-04-25T09:26:40.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:42: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-04-25T09:26:40.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.446386/$cluster-$name.asok' 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:26:40.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:26:40.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:26:40.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:26:40.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:26:40.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:26:40.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:26:40.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:26:40.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:26:40.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:26:40.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-04-25T09:26:40.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:26:40.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d52c9862-d1a2-4180-bb25-545dc5a0abee 2024-04-25T09:26:40.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 d52c9862-d1a2-4180-bb25-545dc5a0abee' 2024-04-25T09:26:40.509 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 d52c9862-d1a2-4180-bb25-545dc5a0abee 2024-04-25T09:26:40.509 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:26:40.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDQISpmF0tXIBAAMuAeJUFm2ygMU2S9JtuEsA== 2024-04-25T09:26:40.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDQISpmF0tXIBAAMuAeJUFm2ygMU2S9JtuEsA=="}' 2024-04-25T09:26:40.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d52c9862-d1a2-4180-bb25-545dc5a0abee -i td/osd-reuse-id/1/new.json 2024-04-25T09:26:41.006 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:26:41.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-04-25T09:26:41.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDQISpmF0tXIBAAMuAeJUFm2ygMU2S9JtuEsA== --osd-uuid d52c9862-d1a2-4180-bb25-545dc5a0abee 2024-04-25T09:26:41.054 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:41.066+0000 7efef126f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:41.091 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:41.102+0000 7efef126f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:41.093 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:41.102+0000 7efef126f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:41.093 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:41.106+0000 7efef126f7c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:26:41.093 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:41.106+0000 7efef126f7c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:26:41.093 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:41.106+0000 7efef126f7c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:26:41.130 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:41.142+0000 7efef126f7c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-04-25T09:26:47.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-04-25T09:26:47.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:26:47.891 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:26:47.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:26:47.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:26:48.452 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:26:48.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:26:48.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:26:48.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:26:48.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:26:48.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:26:48.482 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:48.494+0000 7f2a3012a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:48.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:48.494+0000 7f2a3012a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:48.485 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:48.498+0000 7f2a3012a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:26:48.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:26:48.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:26:48.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:26:48.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:26:48.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:26:48.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:48.889 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:26:48.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:26:48.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:48.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:49.055 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:49.066+0000 7f2a3012a7c0 -1 Falling back to public interface 2024-04-25T09:26:49.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:50.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:50.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:50.283 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:26:50.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:26:50.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:50.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:50.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:51.683 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:26:51.694+0000 7f2a3012a7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:26:51.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:51.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:51.707 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:26:51.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:26:51.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:51.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:52.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:53.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:53.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:53.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:26:53.507 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:26:53.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:53.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:53.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:54.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:54.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:54.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:26:54.986 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:26:54.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:54.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:55.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:56.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:56.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:56.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:26:56.401 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:26:56.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:56.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:56.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:57.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:57.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:57.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:26:58.000 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:26:58.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:58.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:58.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:26:59.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:26:59.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:26:59.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:26:59.412 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:26:59.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:26:59.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:26:59.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:00.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:00.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:00.979 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:27:00.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:27:00.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:00.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:01.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:02.449 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:02.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:02.450 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:27:02.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:27:02.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:02.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:03.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:04.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:04.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:04.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:27:04.079 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:27:04.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:04.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:04.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:05.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:05.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:05.639 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:27:05.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:27:05.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:05.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:06.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:07.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:07.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:07.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:27:07.317 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:27:07.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:07.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:08.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:09.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:09.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:09.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:27:09.218 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:27:09.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:09.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:09.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:10.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:10.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:10.862 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:27:10.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:27:10.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:10.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:12.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:13.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:13.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:13.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:27:13.137 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:27:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:13.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:14.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:15.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:15.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:15.186 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:27:15.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:27:15.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:15.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:15.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:16.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:16.876 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:16.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:27:16.877 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:27:16.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:16.877 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:17.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:18.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:18.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:18.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:27:18.674 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:27:18.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:18.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:19.106 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/110956183,v1:127.0.0.1:6811/110956183] [v2:127.0.0.1:6812/110956183,v1:127.0.0.1:6813/110956183] exists,up d52c9862-d1a2-4180-bb25-545dc5a0abee 2024-04-25T09:27:19.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:27:19.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:27:19.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:27:19.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:43: TEST_reuse_id: create_rbd_pool 2024-04-25T09:27:19.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-04-25T09:27:19.507 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' does not exist 2024-04-25T09:27:19.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-04-25T09:27:19.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-04-25T09:27:20.105 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'rbd' already exists 2024-04-25T09:27:20.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:27:21.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-04-25T09:27:22.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:44: TEST_reuse_id: wait_for_clean 2024-04-25T09:27:22.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:27:22.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:27:22.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:27:22.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:27:22.966 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:27:22.966 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:27:22.966 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:27:22.966 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:27:22.966 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:27:23.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:27:23.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:27:23.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:27:23.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:27:23.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:27:23.079 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:27:23.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:27:23.501 INFO:tasks.workunit.client.0.smithi134.stderr:1' 2024-04-25T09:27:23.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:27:23.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:27:23.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:27:23.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836494 2024-04-25T09:27:23.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836494 2024-04-25T09:27:23.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494' 2024-04-25T09:27:23.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:27:23.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:27:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672968 2024-04-25T09:27:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672968 2024-04-25T09:27:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836494 1-42949672968' 2024-04-25T09:27:23.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:27:23.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836494 2024-04-25T09:27:23.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:27:23.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:27:23.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836494 2024-04-25T09:27:23.825 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:27:23.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836494 2024-04-25T09:27:23.826 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836494 2024-04-25T09:27:23.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-04-25T09:27:23.827 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:27:24.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836494 -lt 21474836494 2024-04-25T09:27:24.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:27:24.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672968 2024-04-25T09:27:24.219 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:27:24.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:27:24.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672968 2024-04-25T09:27:24.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:27:24.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672968 2024-04-25T09:27:24.223 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672968 2024-04-25T09:27:24.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2024-04-25T09:27:24.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:27:24.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672968 -lt 42949672968 2024-04-25T09:27:24.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:27:24.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:27:24.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:27:25.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-04-25T09:27:25.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:27:25.201 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:27:25.201 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:27:25.201 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:27:25.201 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:27:25.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:27:25.202 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:27:25.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-04-25T09:27:25.635 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:27:25.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:27:25.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:27:26.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-04-25T09:27:26.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:27:26.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:27:26.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:45: TEST_reuse_id: destroy_osd td/osd-reuse-id 1 2024-04-25T09:27:26.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/osd-reuse-id 2024-04-25T09:27:26.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=1 2024-04-25T09:27:26.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.1 2024-04-25T09:27:26.851 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already out. 2024-04-25T09:27:26.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/osd-reuse-id TERM osd.1 2024-04-25T09:27:26.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:27:26.870 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:27:26.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:27:26.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:27:26.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:27:28.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:27:28.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.1 2024-04-25T09:27:28.719 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 is already down. 2024-04-25T09:27:28.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.1 --yes-i-really-mean-it 2024-04-25T09:27:29.265 INFO:tasks.workunit.client.0.smithi134.stderr:osd.1 does not exist 2024-04-25T09:27:29.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/osd-reuse-id/1 2024-04-25T09:27:29.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id/1 2024-04-25T09:27:29.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:27:29.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id/1 KILL 2024-04-25T09:27:29.284 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:27:29.284 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:27:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:27:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:27:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:27:29.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:27:29.288 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:27:29.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:27:29.289 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:27:29.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:27:29.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:27:29.291 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:27:29.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:29.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:27:29.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:27:29.293 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:29.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:27:29.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:27:29.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id/1 2024-04-25T09:27:29.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:27:29.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:29.299 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:27:29.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.446386 2024-04-25T09:27:29.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:27:29.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:27:29.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/osd-reuse-id/1 2024-04-25T09:27:29.304 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:46: TEST_reuse_id: run_osd td/osd-reuse-id 1 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/osd-reuse-id 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/osd-reuse-id/1 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none --mon-host=127.0.0.1:7123 ' 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/osd-reuse-id/1' 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/osd-reuse-id/1/journal' 2024-04-25T09:27:29.305 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:27:29.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:27:29.306 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/osd-reuse-id' 2024-04-25T09:27:29.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:27:29.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:27:29.306 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:27:29.306 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:27:29.306 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:29.307 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:27:29.309 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.446386/$cluster-$name.asok' 2024-04-25T09:27:29.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' 2024-04-25T09:27:29.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:27:29.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:27:29.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:27:29.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/osd-reuse-id/$name.log' 2024-04-25T09:27:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/osd-reuse-id/$name.pid' 2024-04-25T09:27:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:27:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:27:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:27:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:27:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:27:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:27:29.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/osd-reuse-id/1 2024-04-25T09:27:29.310 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:27:29.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a2329a12-42c7-4706-b2ca-1b4547eecbee 2024-04-25T09:27:29.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 a2329a12-42c7-4706-b2ca-1b4547eecbee' 2024-04-25T09:27:29.311 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 a2329a12-42c7-4706-b2ca-1b4547eecbee 2024-04-25T09:27:29.312 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:27:29.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQABIipmp9hYFBAAeXfqOMGzjwQtPKKveyHhMQ== 2024-04-25T09:27:29.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQABIipmp9hYFBAAeXfqOMGzjwQtPKKveyHhMQ=="}' 2024-04-25T09:27:29.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a2329a12-42c7-4706-b2ca-1b4547eecbee -i td/osd-reuse-id/1/new.json 2024-04-25T09:27:29.941 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:27:29.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/osd-reuse-id/1/new.json 2024-04-25T09:27:29.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQABIipmp9hYFBAAeXfqOMGzjwQtPKKveyHhMQ== --osd-uuid a2329a12-42c7-4706-b2ca-1b4547eecbee 2024-04-25T09:27:29.991 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:30.005+0000 7f04ddd577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:30.032 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:30.045+0000 7f04ddd577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:30.033 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:30.049+0000 7f04ddd577c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:30.033 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:30.049+0000 7f04ddd577c0 -1 asok(0x55edb5338000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.446386/ceph-osd.1.asok': (2) No such file or directory 2024-04-25T09:27:30.033 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:30.049+0000 7f04ddd577c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:27:30.034 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:30.049+0000 7f04ddd577c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:27:30.034 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:30.049+0000 7f04ddd577c0 -1 bluestore(td/osd-reuse-id/1/block) _read_bdev_label failed to open td/osd-reuse-id/1/block: (2) No such file or directory 2024-04-25T09:27:30.137 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:30.153+0000 7f04ddd577c0 -1 bluestore(td/osd-reuse-id/1) _read_fsid unparsable uuid 2024-04-25T09:27:36.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/osd-reuse-id/1/keyring 2024-04-25T09:27:36.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:27:36.844 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:27:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:27:36.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/osd-reuse-id/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:27:37.420 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:27:37.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:27:37.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=1f2705c9-aa55-4ed9-bacb-896b3e620245 --auth-supported=none --mon-host=127.0.0.1:7123 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-reuse-id/1 --osd-journal=td/osd-reuse-id/1/journal --chdir= --run-dir=td/osd-reuse-id '--admin-socket=/tmp/ceph-asok.446386/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-reuse-id/$name.log' '--pid-file=td/osd-reuse-id/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:27:37.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:27:37.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:27:37.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:27:37.449 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:37.465+0000 7fd9b7a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:37.451 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:37.465+0000 7fd9b7a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:37.453 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:37.469+0000 7fd9b7a5c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:37.453 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:37.469+0000 7fd9b7a5c7c0 -1 asok(0x5627910cc000) AdminSocketConfigObs::init: failed: AdminSocket::bind_and_listen: failed to bind the UNIX domain socket to '/tmp/ceph-asok.446386/ceph-osd.1.asok': (2) No such file or directory 2024-04-25T09:27:37.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:27:37.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:27:37.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:27:37.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:27:37.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:27:37.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:37.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:27:37.830 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:27:37.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:37.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:38.002 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:38.017+0000 7fd9b7a5c7c0 -1 Falling back to public interface 2024-04-25T09:27:38.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:39.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:39.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:39.270 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:27:39.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:27:39.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:39.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:39.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:40.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:40.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:40.676 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:27:40.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:27:40.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:40.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:40.795 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:40.809+0000 7fd9b7a5c7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:27:41.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:42.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:42.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:42.178 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:27:42.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:27:42.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:42.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:27:42.645 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 26 up_thru 26 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1983298985,v1:127.0.0.1:6811/1983298985] [v2:127.0.0.1:6812/1983298985,v1:127.0.0.1:6813/1983298985] exists,up a2329a12-42c7-4706-b2ca-1b4547eecbee 2024-04-25T09:27:42.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-reuse-id.sh:32: run: teardown td/osd-reuse-id 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:27:42.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:27:42.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:27:42.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:27:42.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:27:42.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:27:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:27:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:27:42.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:27:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:27:42.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:27:42.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:42.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:27:42.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:27:42.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:27:42.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:27:42.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:42.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:27:42.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.446386 2024-04-25T09:27:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:27:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:27:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:27:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/osd-reuse-id 0 2024-04-25T09:27:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-reuse-id 2024-04-25T09:27:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:27:42.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-reuse-id KILL 2024-04-25T09:27:42.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:27:42.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:27:42.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:27:42.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:27:42.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:27:42.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:27:42.982 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:27:42.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:27:42.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:27:42.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:27:42.984 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:27:42.985 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:27:42.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:42.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:27:42.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:27:42.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:42.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:27:42.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:27:42.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-reuse-id 2024-04-25T09:27:42.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:27:42.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:42.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.446386 2024-04-25T09:27:42.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.446386 2024-04-25T09:27:42.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:27:42.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:27:42.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:27:42.992 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:27:42.992 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:27:43.044 INFO:tasks.workunit:Running workunit osd/pg-split-merge.sh... 2024-04-25T09:27:43.045 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/pg-split-merge.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh 2024-04-25T09:27:43.100 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:27:43.106 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:27:43.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:27:43.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:27:43.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:27:43.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/pg-split-merge 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:5: run: local dir=td/pg-split-merge 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:6: run: shift 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: export CEPH_MON=127.0.0.1:7147 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:8: run: CEPH_MON=127.0.0.1:7147 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:9: run: export CEPH_ARGS 2024-04-25T09:27:43.107 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: uuidgen 2024-04-25T09:27:43.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:10: run: CEPH_ARGS+='--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none ' 2024-04-25T09:27:43.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:11: run: CEPH_ARGS+='--mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:27:43.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: set 2024-04-25T09:27:43.108 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:27:43.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:13: run: local 'funcs=TEST_a_merge_empty 2024-04-25T09:27:43.111 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_import_after_merge_and_gap 2024-04-25T09:27:43.111 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_import_after_split' 2024-04-25T09:27:43.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:27:43.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:27:43.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:27:43.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:27:43.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:27:43.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:27:43.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:27:43.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:27:43.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:27:43.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:27:43.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:27:43.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:27:43.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:27:43.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:27:43.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:27:43.115 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:27:43.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:27:43.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:27:43.116 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:27:43.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:43.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:27:43.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:27:43.118 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:27:43.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:27:43.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:27:43.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:27:43.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:27:43.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:43.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:27:43.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451985 2024-04-25T09:27:43.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:27:43.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:27:43.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:27:43.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:27:43.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:43.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:27:43.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.451985 2024-04-25T09:27:43.126 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:27:43.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_a_merge_empty td/pg-split-merge 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:22: TEST_a_merge_empty: local dir=td/pg-split-merge 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:24: TEST_a_merge_empty: run_mon td/pg-split-merge a --osd_pool_default_size=3 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:27:43.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=3 2024-04-25T09:27:43.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:27:43.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:27:43.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:27:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:27:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:43.611 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:27:43.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:27:43.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-04-25T09:27:43.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:27:43.848 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:27:43.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:27:43.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:27:43.849 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:27:43.850 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:27:43.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:27:43.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:27:43.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:27:43.850 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:27:43.850 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:43.850 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:27:43.850 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451985/ceph-mon.a.asok 2024-04-25T09:27:43.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:27:43.851 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451985/ceph-mon.a.asok config get fsid 2024-04-25T09:27:43.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:27:43.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:27:43.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:27:43.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:27:43.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:27:43.962 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:27:43.963 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:27:43.963 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:27:43.963 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:27:43.963 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:43.963 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:27:43.963 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451985/ceph-mon.a.asok 2024-04-25T09:27:43.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:27:43.963 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451985/ceph-mon.a.asok config get mon_host 2024-04-25T09:27:44.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:25: TEST_a_merge_empty: run_mgr td/pg-split-merge x 2024-04-25T09:27:44.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:27:44.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:27:44.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:27:44.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:27:44.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:27:44.074 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:27:45.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:27:45.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:27:45.002 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:27:45.002 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:27:45.003 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:45.003 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:27:45.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:27:45.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:27:45.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:27:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:26: TEST_a_merge_empty: run_osd td/pg-split-merge 0 2024-04-25T09:27:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:27:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:27:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:27:45.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:27:45.045 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:27:45.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:27:45.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:27:45.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:27:45.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:27:45.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:27:45.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=30b8cdc8-e662-459d-b567-532a65250722 2024-04-25T09:27:45.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 30b8cdc8-e662-459d-b567-532a65250722' 2024-04-25T09:27:45.047 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 30b8cdc8-e662-459d-b567-532a65250722 2024-04-25T09:27:45.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:27:45.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQARIipmkJW5BBAAvvkZApNYmItzQuzZLdKmNw== 2024-04-25T09:27:45.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQARIipmkJW5BBAAvvkZApNYmItzQuzZLdKmNw=="}' 2024-04-25T09:27:45.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 30b8cdc8-e662-459d-b567-532a65250722 -i td/pg-split-merge/0/new.json 2024-04-25T09:27:45.384 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:27:45.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:27:45.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQARIipmkJW5BBAAvvkZApNYmItzQuzZLdKmNw== --osd-uuid 30b8cdc8-e662-459d-b567-532a65250722 2024-04-25T09:27:45.432 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:45.445+0000 7f7eebfe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:45.444 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:45.457+0000 7f7eebfe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:45.445 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:45.461+0000 7f7eebfe47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:45.446 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:45.461+0000 7f7eebfe47c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:27:45.446 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:45.461+0000 7f7eebfe47c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:27:45.446 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:45.461+0000 7f7eebfe47c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:27:45.484 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:45.497+0000 7f7eebfe47c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:27:52.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:27:52.035 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:27:52.036 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:27:52.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:27:52.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:27:52.603 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:27:52.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:27:52.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:27:52.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:27:52.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:27:52.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:27:52.636 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:52.648+0000 7fd6a57937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:52.638 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:52.652+0000 7fd6a57937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:52.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:52.652+0000 7fd6a57937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:27:53.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:27:53.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:27:53.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:27:53.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:27:53.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:27:53.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:53.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:27:53.069 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:27:53.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:53.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:53.198 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:53.212+0000 7fd6a57937c0 -1 Falling back to public interface 2024-04-25T09:27:53.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:54.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:54.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:54.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:27:54.466 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:27:54.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:54.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:54.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:55.766 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:27:55.780+0000 7fd6a57937c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:27:55.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:55.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:55.883 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:27:55.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:27:55.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:55.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:56.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:57.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:57.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:57.504 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:27:57.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:27:57.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:57.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:57.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:27:58.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:27:58.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:27:58.957 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:27:58.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:27:58.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:27:58.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:27:59.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:00.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:00.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:00.381 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:28:00.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:28:00.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:00.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:00.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:01.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:01.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:01.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:28:01.892 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:28:01.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:01.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:02.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:03.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:03.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:03.314 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:28:03.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:28:03.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:03.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:03.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:04.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:04.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:04.781 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:28:04.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:28:04.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:04.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:05.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:06.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:06.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:06.249 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:28:06.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:28:06.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:06.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:06.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:07.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:07.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:07.727 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:28:07.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:28:07.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:07.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:08.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:09.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:09.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:09.212 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:28:09.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:28:09.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:09.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:10.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:11.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:11.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:11.258 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:28:11.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:28:11.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:11.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:12.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:13.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:13.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:13.104 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:28:13.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:28:13.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:13.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:13.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:14.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:14.936 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:14.936 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:28:14.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:28:14.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:14.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:15.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:16.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:16.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:16.774 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:28:16.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:28:16.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:16.775 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:17.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:18.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:18.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:18.608 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:28:18.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:28:18.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:18.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:19.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:20.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:20.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:20.825 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:28:20.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:28:20.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:20.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:28:21.370 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/306933903,v1:127.0.0.1:6803/306933903] [v2:127.0.0.1:6804/306933903,v1:127.0.0.1:6805/306933903] exists,up 30b8cdc8-e662-459d-b567-532a65250722 2024-04-25T09:28:21.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:28:21.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:28:21.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:28:21.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:27: TEST_a_merge_empty: run_osd td/pg-split-merge 1 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/1 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/1' 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/1/journal' 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:28:21.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:28:21.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:28:21.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:28:21.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:28:21.374 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:28:21.374 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:28:21.374 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:28:21.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:28:21.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:28:21.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:28:21.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/1 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f58512cb-fa4f-4483-aa25-430e34c74915 2024-04-25T09:28:21.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f58512cb-fa4f-4483-aa25-430e34c74915' 2024-04-25T09:28:21.377 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 f58512cb-fa4f-4483-aa25-430e34c74915 2024-04-25T09:28:21.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:28:21.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1IipmGjGmGBAABrI7ey8fVjmdaIMmJladWg== 2024-04-25T09:28:21.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1IipmGjGmGBAABrI7ey8fVjmdaIMmJladWg=="}' 2024-04-25T09:28:21.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f58512cb-fa4f-4483-aa25-430e34c74915 -i td/pg-split-merge/1/new.json 2024-04-25T09:28:21.999 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:28:22.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/1/new.json 2024-04-25T09:28:22.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1IipmGjGmGBAABrI7ey8fVjmdaIMmJladWg== --osd-uuid f58512cb-fa4f-4483-aa25-430e34c74915 2024-04-25T09:28:22.050 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:22.064+0000 7fa11e0517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:22.079 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:22.092+0000 7fa11e0517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:22.080 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:22.092+0000 7fa11e0517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:22.080 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:22.092+0000 7fa11e0517c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:28:22.080 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:22.092+0000 7fa11e0517c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:28:22.081 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:22.092+0000 7fa11e0517c0 -1 bluestore(td/pg-split-merge/1/block) _read_bdev_label failed to open td/pg-split-merge/1/block: (2) No such file or directory 2024-04-25T09:28:22.171 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:22.184+0000 7fa11e0517c0 -1 bluestore(td/pg-split-merge/1) _read_fsid unparsable uuid 2024-04-25T09:28:29.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/1/keyring 2024-04-25T09:28:29.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:28:29.465 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:28:29.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:28:29.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:28:30.033 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:28:30.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:28:30.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/1 --osd-journal=td/pg-split-merge/1/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:28:30.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:28:30.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:28:30.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:28:30.066 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:30.080+0000 7f8bddee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:30.093 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:30.108+0000 7f8bddee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:30.094 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:30.108+0000 7f8bddee47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:28:30.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:28:30.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:28:30.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:28:30.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:28:30.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:28:30.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:30.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:28:30.457 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:28:30.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:30.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:30.650 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:30.663+0000 7f8bddee47c0 -1 Falling back to public interface 2024-04-25T09:28:30.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:31.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:31.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:31.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:28:31.913 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:28:31.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:31.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:32.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:33.276 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:28:33.287+0000 7f8bddee47c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:28:33.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:33.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:33.330 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:28:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:28:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:33.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:34.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:35.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:35.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:35.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:28:35.086 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:28:35.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:35.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:35.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:36.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:36.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:36.603 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:28:36.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:28:36.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:36.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:37.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:38.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:38.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:38.045 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:28:38.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:28:38.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:38.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:38.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:39.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:39.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:39.550 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:28:39.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:28:39.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:39.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:40.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:41.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:41.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:41.257 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:28:41.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:28:41.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:41.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:41.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:42.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:42.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:42.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:28:42.705 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:28:42.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:42.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:43.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:44.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:44.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:44.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:28:44.150 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:28:44.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:44.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:44.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:45.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:45.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:45.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:28:45.710 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:28:45.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:45.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:46.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:47.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:47.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:47.317 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:28:47.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:28:47.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:47.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:47.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:48.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:48.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:48.741 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:28:48.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:28:48.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:48.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:49.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:50.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:50.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:50.483 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:28:50.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:28:50.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:50.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:51.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:52.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:52.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:52.123 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:28:52.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:28:52.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:52.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:53.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:54.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:54.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:54.196 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:28:54.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:28:54.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:54.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:55.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:56.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:56.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:56.265 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:28:56.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:28:56.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:56.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:57.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:58.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:28:58.005 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:28:58.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:28:58.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:58.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:28:59.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:28:59.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:28:59.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:28:59.829 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:28:59.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:28:59.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:00.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:01.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:01.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:01.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:29:01.256 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T09:29:01.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:01.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:29:01.681 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1068942997,v1:127.0.0.1:6811/1068942997] [v2:127.0.0.1:6812/1068942997,v1:127.0.0.1:6813/1068942997] exists,up f58512cb-fa4f-4483-aa25-430e34c74915 2024-04-25T09:29:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:29:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:29:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:29:01.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:28: TEST_a_merge_empty: run_osd td/pg-split-merge 2 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/2 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:29:01.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:29:01.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:29:01.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:29:01.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:29:01.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:29:01.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:29:01.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:29:01.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:29:01.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/2 2024-04-25T09:29:01.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:29:01.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=23ed7c75-d659-4a87-ae47-7e16d39179c8 2024-04-25T09:29:01.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 23ed7c75-d659-4a87-ae47-7e16d39179c8' 2024-04-25T09:29:01.688 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 23ed7c75-d659-4a87-ae47-7e16d39179c8 2024-04-25T09:29:01.688 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:29:01.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBdIipmgYAUKxAAZ+gTwk4O6Ruq7072NNpf+g== 2024-04-25T09:29:01.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBdIipmgYAUKxAAZ+gTwk4O6Ruq7072NNpf+g=="}' 2024-04-25T09:29:01.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 23ed7c75-d659-4a87-ae47-7e16d39179c8 -i td/pg-split-merge/2/new.json 2024-04-25T09:29:02.155 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:29:02.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/2/new.json 2024-04-25T09:29:02.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBdIipmgYAUKxAAZ+gTwk4O6Ruq7072NNpf+g== --osd-uuid 23ed7c75-d659-4a87-ae47-7e16d39179c8 2024-04-25T09:29:02.208 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:02.219+0000 7fcef90437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:02.234 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:02.247+0000 7fcef90437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:02.235 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:02.247+0000 7fcef90437c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:02.235 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:02.247+0000 7fcef90437c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:29:02.235 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:02.247+0000 7fcef90437c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:29:02.235 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:02.247+0000 7fcef90437c0 -1 bluestore(td/pg-split-merge/2/block) _read_bdev_label failed to open td/pg-split-merge/2/block: (2) No such file or directory 2024-04-25T09:29:02.273 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:02.287+0000 7fcef90437c0 -1 bluestore(td/pg-split-merge/2) _read_fsid unparsable uuid 2024-04-25T09:29:09.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/2/keyring 2024-04-25T09:29:09.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:29:09.244 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:29:09.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:29:09.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:29:09.803 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:29:09.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:29:09.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:29:09.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:29:09.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:29:09.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:29:09.836 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:09.851+0000 7f0d0175a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:09.841 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:09.855+0000 7f0d0175a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:09.842 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:09.855+0000 7f0d0175a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:29:10.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:29:10.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:29:10.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:29:10.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:29:10.229 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:29:10.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:29:10.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:10.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:29:10.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:10.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:10.394 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:10.407+0000 7f0d0175a7c0 -1 Falling back to public interface 2024-04-25T09:29:10.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:11.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:11.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:11.662 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:29:11.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:29:11.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:11.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:12.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:12.997 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:29:13.011+0000 7f0d0175a7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:29:13.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:13.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:13.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:29:13.122 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:29:13.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:13.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:13.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:14.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:14.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:14.635 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:29:14.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:29:14.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:14.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:15.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:16.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:16.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:16.171 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:29:16.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:29:16.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:16.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:16.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:17.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:17.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:17.842 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:29:17.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:29:17.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:17.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:18.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:19.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:19.269 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:29:19.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:19.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:29:19.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:19.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:19.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:20.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:20.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:20.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:29:20.739 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:29:20.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:20.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:21.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:22.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:22.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:22.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:29:22.275 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:29:22.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:22.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:23.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:24.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:24.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:24.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:29:24.006 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:29:24.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:24.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:25.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:25.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:25.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:29:25.479 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:29:25.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:25.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:25.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:26.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:26.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:26.897 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:29:26.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:29:26.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:26.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:27.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:28.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:28.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:28.480 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:29:28.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:29:28.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:28.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:29.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:30.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:30.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:30.336 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:29:30.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:29:30.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:30.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:31.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:32.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:32.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:32.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:29:32.102 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:29:32.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:32.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:32.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:33.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:33.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:33.699 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:29:33.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:29:33.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:33.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:34.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:35.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:35.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:35.438 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:29:35.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:29:35.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:35.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:35.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:36.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:36.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:36.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:29:36.971 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:29:36.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:36.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:37.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:38.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:38.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:38.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:29:38.766 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:29:38.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:38.767 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:39.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:29:40.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:29:40.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:29:40.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:29:40.411 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T09:29:40.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:29:40.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:29:40.840 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2400227560,v1:127.0.0.1:6819/2400227560] [v2:127.0.0.1:6820/2400227560,v1:127.0.0.1:6821/2400227560] exists,up 23ed7c75-d659-4a87-ae47-7e16d39179c8 2024-04-25T09:29:40.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:29:40.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:29:40.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:29:40.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:30: TEST_a_merge_empty: ceph osd pool create foo 2 2024-04-25T09:29:41.531 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T09:29:41.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:31: TEST_a_merge_empty: ceph osd pool set foo pgp_num 1 2024-04-25T09:29:43.114 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 pgp_num to 1 2024-04-25T09:29:43.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:33: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:29:43.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:29:43.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:29:43.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:29:43.136 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:29:43.137 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:29:43.137 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:29:43.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:29:43.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:29:43.138 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:29:43.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:29:43.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:29:43.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:29:43.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:29:43.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:29:43.247 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:29:43.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:29:43.896 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:29:43.896 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:29:43.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:29:43.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:43.897 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:29:44.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836509 2024-04-25T09:29:44.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836509 2024-04-25T09:29:44.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509' 2024-04-25T09:29:44.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:44.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:29:44.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672978 2024-04-25T09:29:44.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672978 2024-04-25T09:29:44.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672978' 2024-04-25T09:29:44.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:44.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:29:44.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509450 2024-04-25T09:29:44.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509450 2024-04-25T09:29:44.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836509 1-42949672978 2-64424509450' 2024-04-25T09:29:44.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:44.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836509 2024-04-25T09:29:44.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:44.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:29:44.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836509 2024-04-25T09:29:44.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:44.407 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836509 2024-04-25T09:29:44.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836509 2024-04-25T09:29:44.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2024-04-25T09:29:44.407 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:29:44.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836507 -lt 21474836509 2024-04-25T09:29:44.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:29:45.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:29:45.804 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:29:46.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836509 -lt 21474836509 2024-04-25T09:29:46.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:46.526 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:46.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672978 2024-04-25T09:29:46.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:29:46.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672978 2024-04-25T09:29:46.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:46.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672978 2024-04-25T09:29:46.530 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672978 2024-04-25T09:29:46.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2024-04-25T09:29:46.530 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:29:46.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672978 -lt 42949672978 2024-04-25T09:29:46.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:46.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:46.972 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509450 2024-04-25T09:29:46.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:29:46.974 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509450 2024-04-25T09:29:46.974 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:46.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509450 2024-04-25T09:29:46.975 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509450 2024-04-25T09:29:46.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-04-25T09:29:46.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:29:47.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509450 -lt 64424509450 2024-04-25T09:29:47.501 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:29:47.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:29:47.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:29:48.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:29:48.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:29:48.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:29:48.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:29:48.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:29:48.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:29:48.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:29:48.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:29:48.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:29:48.494 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:29:48.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:29:48.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:29:49.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:29:49.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:29:49.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:29:49.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:37: TEST_a_merge_empty: rados -p foo put foo1 /etc/passwd 2024-04-25T09:29:49.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:38: TEST_a_merge_empty: rados -p foo put foo2 /etc/passwd 2024-04-25T09:29:49.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:39: TEST_a_merge_empty: rados -p foo put foo3 /etc/passwd 2024-04-25T09:29:49.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:40: TEST_a_merge_empty: rados -p foo put foo4 /etc/passwd 2024-04-25T09:29:49.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:42: TEST_a_merge_empty: rados -p foo put foo5 /etc/passwd 2024-04-25T09:29:49.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:43: TEST_a_merge_empty: rados -p foo put foo6 /etc/passwd 2024-04-25T09:29:50.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:44: TEST_a_merge_empty: rados -p foo put foo8 /etc/passwd 2024-04-25T09:29:50.379 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:45: TEST_a_merge_empty: rados -p foo put foo10 /etc/passwd 2024-04-25T09:29:50.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:46: TEST_a_merge_empty: rados -p foo put foo11 /etc/passwd 2024-04-25T09:29:50.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:47: TEST_a_merge_empty: rados -p foo put foo12 /etc/passwd 2024-04-25T09:29:50.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:48: TEST_a_merge_empty: rados -p foo put foo16 /etc/passwd 2024-04-25T09:29:51.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:50: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:29:51.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:29:51.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:29:51.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:29:51.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:29:51.082 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:29:51.082 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:29:51.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:29:51.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:29:51.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:29:51.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:29:51.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:29:51.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:29:51.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:29:51.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:29:51.199 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:29:51.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:29:51.626 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:29:51.627 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:29:51.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:29:51.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:51.627 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:29:51.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836512 2024-04-25T09:29:51.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836512 2024-04-25T09:29:51.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512' 2024-04-25T09:29:51.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:51.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:29:51.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672981 2024-04-25T09:29:51.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672981 2024-04-25T09:29:51.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672981' 2024-04-25T09:29:51.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:51.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:29:52.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509454 2024-04-25T09:29:52.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509454 2024-04-25T09:29:52.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836512 1-42949672981 2-64424509454' 2024-04-25T09:29:52.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:52.109 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836512 2024-04-25T09:29:52.109 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:52.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:29:52.110 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836512 2024-04-25T09:29:52.110 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:52.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836512 2024-04-25T09:29:52.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2024-04-25T09:29:52.112 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836512 2024-04-25T09:29:52.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:29:52.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836510 -lt 21474836512 2024-04-25T09:29:52.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:29:53.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:29:53.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:29:53.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836513 -lt 21474836512 2024-04-25T09:29:53.937 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:53.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672981 2024-04-25T09:29:53.938 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:53.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:29:53.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672981 2024-04-25T09:29:53.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:53.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672981 2024-04-25T09:29:53.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672981' 2024-04-25T09:29:53.942 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672981 2024-04-25T09:29:53.943 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:29:54.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672982 -lt 42949672981 2024-04-25T09:29:54.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:54.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509454 2024-04-25T09:29:54.401 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:54.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:29:54.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509454 2024-04-25T09:29:54.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:54.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509454 2024-04-25T09:29:54.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2024-04-25T09:29:54.405 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509454 2024-04-25T09:29:54.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:29:54.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509454 -lt 64424509454 2024-04-25T09:29:54.816 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:29:54.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:29:54.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:29:55.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:29:55.382 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:29:55.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:29:55.382 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:29:55.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:29:55.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:29:55.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:29:55.383 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:29:55.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:29:55.809 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:29:55.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:29:55.810 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:29:56.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:29:56.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:29:56.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:29:56.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:52: TEST_a_merge_empty: ceph tell osd.1 config set osd_debug_no_purge_strays true 2024-04-25T09:29:56.517 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T09:29:56.517 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "" 2024-04-25T09:29:56.517 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T09:29:56.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:53: TEST_a_merge_empty: ceph osd pool set foo size 2 2024-04-25T09:29:57.354 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 2 2024-04-25T09:29:57.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:54: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:29:57.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:29:57.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:29:57.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:29:57.378 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:29:57.379 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:29:57.379 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:29:57.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:29:57.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:29:57.380 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:29:57.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:29:57.492 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:29:57.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:29:57.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:29:57.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:29:57.493 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:29:58.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:29:58.123 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:29:58.123 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:29:58.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:29:58.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:58.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:29:58.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836516 2024-04-25T09:29:58.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836516 2024-04-25T09:29:58.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516' 2024-04-25T09:29:58.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:58.283 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:29:58.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672985 2024-04-25T09:29:58.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672985 2024-04-25T09:29:58.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672985' 2024-04-25T09:29:58.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:29:58.541 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:29:58.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509457 2024-04-25T09:29:58.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509457 2024-04-25T09:29:58.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836516 1-42949672985 2-64424509457' 2024-04-25T09:29:58.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:29:58.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836516 2024-04-25T09:29:58.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:29:58.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:29:58.706 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836516 2024-04-25T09:29:58.706 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:29:58.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836516 2024-04-25T09:29:58.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836516' 2024-04-25T09:29:58.707 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836516 2024-04-25T09:29:58.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:29:59.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836514 -lt 21474836516 2024-04-25T09:29:59.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:30:00.117 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:30:00.118 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:30:00.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836516 -lt 21474836516 2024-04-25T09:30:00.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:30:00.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672985 2024-04-25T09:30:00.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:30:00.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:30:00.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672985 2024-04-25T09:30:00.536 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:30:00.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672985 2024-04-25T09:30:00.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-04-25T09:30:00.538 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672985 2024-04-25T09:30:00.538 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:30:00.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672985 -lt 42949672985 2024-04-25T09:30:00.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:30:00.976 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509457 2024-04-25T09:30:00.977 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:30:00.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:30:00.978 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509457 2024-04-25T09:30:00.979 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:30:00.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509457 2024-04-25T09:30:00.980 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509457 2024-04-25T09:30:00.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509457' 2024-04-25T09:30:00.980 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:30:01.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509457 2024-04-25T09:30:01.408 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:30:01.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:30:01.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:30:01.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:30:01.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:30:01.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:30:01.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:30:01.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:30:01.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:30:01.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:30:01.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:30:02.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:30:02.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:30:02.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:30:02.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:30:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:30:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:30:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:30:03.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:56: TEST_a_merge_empty: kill_daemons td/pg-split-merge TERM osd.2 2024-04-25T09:30:03.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:30:03.007 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:30:03.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:30:03.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:30:03.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:30:03.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:30:03.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:57: TEST_a_merge_empty: ceph-objectstore-tool --data-path td/pg-split-merge/2 --op remove --pgid 1.1 --force 2024-04-25T09:30:04.715 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T09:30:04.715 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T09:30:04.715 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:30:04.815 INFO:tasks.workunit.client.0.smithi134.stdout:Remove successful 2024-04-25T09:30:05.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:58: TEST_a_merge_empty: activate_osd td/pg-split-merge 2 2024-04-25T09:30:05.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:30:05.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:30:05.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T09:30:05.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:30:05.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/2 2024-04-25T09:30:05.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/2' 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/2/journal' 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:30:05.253 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:30:05.254 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:30:05.254 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:30:05.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/2 2024-04-25T09:30:05.256 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:30:05.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T09:30:05.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/2 --osd-journal=td/pg-split-merge/2/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:30:05.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/2/whoami 2024-04-25T09:30:05.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T09:30:05.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:30:05.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:30:05.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:30:05.291 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:05.305+0000 7faa93e2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:05.294 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:05.309+0000 7faa93e2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:05.295 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:05.309+0000 7faa93e2f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:05.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T09:30:05.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:30:05.752 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:30:05.752 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:30:05.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:30:05.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:30:05.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:05.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:30:05.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:05.753 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:05.842 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:05.857+0000 7faa93e2f7c0 -1 Falling back to public interface 2024-04-25T09:30:06.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:07.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:07.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:07.229 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:30:07.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:30:07.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:07.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:07.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:08.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:08.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:08.634 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:30:08.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:30:08.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:08.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:08.768 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:08.781+0000 7faa93e2f7c0 -1 osd.2 27 log_to_monitors true 2024-04-25T09:30:09.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:10.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:10.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:10.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:30:10.173 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:30:10.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:10.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:30:10.592 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 30 up_thru 17 down_at 28 last_clean_interval [15,27) [v2:127.0.0.1:6818/1535677429,v1:127.0.0.1:6819/1535677429] [v2:127.0.0.1:6820/1535677429,v1:127.0.0.1:6821/1535677429] exists,up 23ed7c75-d659-4a87-ae47-7e16d39179c8 2024-04-25T09:30:10.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:30:10.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:30:10.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:30:10.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:60: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:30:10.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:30:10.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:30:10.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:30:10.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:30:10.594 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:30:10.594 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:30:10.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:30:10.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:30:10.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:30:10.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:30:10.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:30:10.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:30:10.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:30:10.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:30:10.699 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:30:11.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:30:11.130 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:30:11.130 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:30:11.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:30:11.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:30:11.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:30:11.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836520 2024-04-25T09:30:11.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836520 2024-04-25T09:30:11.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520' 2024-04-25T09:30:11.308 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:30:11.308 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:30:11.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672989 2024-04-25T09:30:11.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672989 2024-04-25T09:30:11.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-42949672989' 2024-04-25T09:30:11.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:30:11.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:30:11.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018883 2024-04-25T09:30:11.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018883 2024-04-25T09:30:11.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-42949672989 2-128849018883' 2024-04-25T09:30:11.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:30:11.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836520 2024-04-25T09:30:11.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:30:11.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:30:11.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836520 2024-04-25T09:30:11.618 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:30:11.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836520 2024-04-25T09:30:11.620 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836520 2024-04-25T09:30:11.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-04-25T09:30:11.620 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:30:12.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836520 -lt 21474836520 2024-04-25T09:30:12.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:30:12.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672989 2024-04-25T09:30:12.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:30:12.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:30:12.032 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672989 2024-04-25T09:30:12.033 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:30:12.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672989 2024-04-25T09:30:12.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672989' 2024-04-25T09:30:12.034 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672989 2024-04-25T09:30:12.034 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:30:12.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672989 -lt 42949672989 2024-04-25T09:30:12.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:30:12.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-128849018883 2024-04-25T09:30:12.458 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:30:12.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:30:12.459 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-128849018883 2024-04-25T09:30:12.459 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:30:12.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018883 2024-04-25T09:30:12.461 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 128849018883 2024-04-25T09:30:12.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 128849018883' 2024-04-25T09:30:12.461 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:30:12.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018883 -lt 128849018883 2024-04-25T09:30:12.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:30:12.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:30:12.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:30:13.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:30:13.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:30:13.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:30:13.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:30:13.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:30:13.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:30:13.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:30:13.384 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:30:13.840 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:30:13.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:30:13.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:30:13.841 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:30:14.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:30:14.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:30:14.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:30:14.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:66: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change true 2024-04-25T09:30:14.579 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T09:30:14.579 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "" 2024-04-25T09:30:14.579 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T09:30:14.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:67: TEST_a_merge_empty: ceph osd out 0 1 2024-04-25T09:30:15.134 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already out. osd.1 is already out. 2024-04-25T09:30:15.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:68: TEST_a_merge_empty: ceph osd pool set foo pg_num 1 2024-04-25T09:30:16.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:69: TEST_a_merge_empty: sleep 5 2024-04-25T09:30:21.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:70: TEST_a_merge_empty: ceph tell osd.2 config set osd_debug_no_acting_change false 2024-04-25T09:30:21.211 INFO:tasks.workunit.client.0.smithi134.stdout:{ 2024-04-25T09:30:21.211 INFO:tasks.workunit.client.0.smithi134.stdout: "success": "" 2024-04-25T09:30:21.212 INFO:tasks.workunit.client.0.smithi134.stdout:} 2024-04-25T09:30:21.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:74: TEST_a_merge_empty: ceph osd in 0 1 2024-04-25T09:30:21.747 INFO:tasks.workunit.client.0.smithi134.stderr:osd.0 is already in. osd.1 is already in. 2024-04-25T09:30:21.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:75: TEST_a_merge_empty: ceph osd pool set foo size 3 2024-04-25T09:30:23.238 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 3 2024-04-25T09:30:23.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:77: TEST_a_merge_empty: wait_for_clean 2024-04-25T09:30:23.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:30:23.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:30:23.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:30:23.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:30:23.261 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:30:23.261 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:30:23.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:30:23.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:30:23.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:30:23.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:30:23.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:30:23.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:30:23.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:30:23.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:30:23.374 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:30:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:30:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:30:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:30:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:30:24.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:30:24.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:30:24.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836525 2024-04-25T09:30:24.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836525 2024-04-25T09:30:24.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836525' 2024-04-25T09:30:24.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:30:24.165 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:30:24.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949672994 2024-04-25T09:30:24.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949672994 2024-04-25T09:30:24.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836525 1-42949672994' 2024-04-25T09:30:24.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:30:24.404 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:30:24.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018888 2024-04-25T09:30:24.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018888 2024-04-25T09:30:24.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836525 1-42949672994 2-128849018888' 2024-04-25T09:30:24.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:30:24.562 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836525 2024-04-25T09:30:24.562 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:30:24.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:30:24.563 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836525 2024-04-25T09:30:24.564 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:30:24.564 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836525 2024-04-25T09:30:24.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836525 2024-04-25T09:30:24.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2024-04-25T09:30:24.565 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:30:24.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836523 -lt 21474836525 2024-04-25T09:30:24.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:30:25.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:30:25.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:30:26.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836525 -lt 21474836525 2024-04-25T09:30:26.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:30:26.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949672994 2024-04-25T09:30:26.495 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:30:26.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:30:26.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949672994 2024-04-25T09:30:26.498 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:30:26.499 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949672994 2024-04-25T09:30:26.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949672994 2024-04-25T09:30:26.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949672994' 2024-04-25T09:30:26.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:30:26.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949672994 -lt 42949672994 2024-04-25T09:30:26.909 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:30:26.909 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-128849018888 2024-04-25T09:30:26.910 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:30:26.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:30:26.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-128849018888 2024-04-25T09:30:26.911 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:30:26.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018888 2024-04-25T09:30:26.913 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 128849018888 2024-04-25T09:30:26.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 128849018888' 2024-04-25T09:30:26.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:30:27.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018888 -lt 128849018888 2024-04-25T09:30:27.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:30:27.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:30:27.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:30:27.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:30:27.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:30:27.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:30:27.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:30:27.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:30:27.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:30:27.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:30:27.940 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:30:28.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:30:28.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:30:28.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:30:28.341 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:30:28.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:30:28.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:30:28.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:30:28.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:81: TEST_a_merge_empty: ceph pg scrub 1.0 2024-04-25T09:30:29.228 INFO:tasks.workunit.client.0.smithi134.stderr:instructing pg 1.0 on osd.1 to scrub 2024-04-25T09:30:29.244 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:82: TEST_a_merge_empty: sleep 10 2024-04-25T09:30:39.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:83: TEST_a_merge_empty: ceph log last debug 2024-04-25T09:30:39.805 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:21.643894+0000 mgr.x (mgr.4100) 76 : cluster [DBG] pgmap v103: 2 pgs: 2 unknown; 82 KiB data, 27 MiB used, 100 GiB / 100 GiB avail 2024-04-25T09:30:39.805 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:22.403467+0000 mon.a (mon.0) 418 : cluster [DBG] osdmap e38: 3 total, 3 up, 3 in 2024-04-25T09:30:39.805 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:22.846618+0000 mon.a (mon.0) 420 : cluster [DBG] osdmap e39: 3 total, 3 up, 3 in 2024-04-25T09:30:39.805 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:23.341084+0000 mon.a (mon.0) 423 : cluster [DBG] osdmap e40: 3 total, 3 up, 3 in 2024-04-25T09:30:39.805 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:23.795231+0000 mon.a (mon.0) 424 : cluster [DBG] osdmap e41: 3 total, 3 up, 3 in 2024-04-25T09:30:39.805 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:23.644421+0000 mgr.x (mgr.4100) 77 : cluster [DBG] pgmap v108: 1 pgs: 1 unknown; 82 KiB data, 81 MiB used, 300 GiB / 300 GiB avail 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:24.052888+0000 mon.a (mon.0) 427 : cluster [DBG] osdmap e42: 3 total, 3 up, 3 in 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:24.430495+0000 mon.a (mon.0) 428 : cluster [DBG] osdmap e43: 3 total, 3 up, 3 in 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:24.551911+0000 osd.1 (osd.1) 3 : cluster [INF] 1.0 continuing backfill to osd.2 from (24'7,24'7] MIN to 24'7 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:25.420806+0000 mon.a (mon.0) 431 : cluster [DBG] osdmap e44: 3 total, 3 up, 3 in 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:25.839846+0000 mon.a (mon.0) 432 : cluster [DBG] osdmap e45: 3 total, 3 up, 3 in 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:25.644943+0000 mgr.x (mgr.4100) 78 : cluster [DBG] pgmap v113: 1 pgs: 1 remapped+peering; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 21 KiB/s wr, 1 op/s 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:27.645479+0000 mgr.x (mgr.4100) 79 : cluster [DBG] pgmap v115: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 12 KiB/s wr, 1 op/s; 12 KiB/s, 1 objects/s recovering 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:29.646065+0000 mgr.x (mgr.4100) 84 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 8.6 KiB/s wr, 0 op/s; 8.4 KiB/s, 0 objects/s recovering 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:29.742063+0000 osd.1 (osd.1) 4 : cluster [DBG] 1.0 scrub starts 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:29.747370+0000 osd.1 (osd.1) 5 : cluster [DBG] 1.0 scrub ok 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:31.646618+0000 mgr.x (mgr.4100) 85 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 6.5 KiB/s, 0 objects/s recovering 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:33.647189+0000 mgr.x (mgr.4100) 86 : cluster [DBG] pgmap v118: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 5.7 KiB/s, 0 objects/s recovering 2024-04-25T09:30:39.806 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:35.647705+0000 mgr.x (mgr.4100) 87 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.7 KiB/s, 0 objects/s recovering 2024-04-25T09:30:39.807 INFO:tasks.workunit.client.0.smithi134.stdout:2024-04-25T09:30:37.648217+0000 mgr.x (mgr.4100) 88 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 130 KiB data, 81 MiB used, 300 GiB / 300 GiB avail; 4.0 KiB/s, 0 objects/s recovering 2024-04-25T09:30:39.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:84: TEST_a_merge_empty: ceph pg ls 2024-04-25T09:30:40.211 INFO:tasks.workunit.client.0.smithi134.stdout:PG OBJECTS DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG LOG_DUPS STATE SINCE VERSION REPORTED UP ACTING SCRUB_STAMP DEEP_SCRUB_STAMP LAST_SCRUB_DURATION SCRUB_SCHEDULING 2024-04-25T09:30:40.211 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 11 0 0 0 132748 0 0 0 11 active+clean 10s 24'7 45:95 [1,0,2]p1 [1,0,2]p1 2024-04-25T09:30:29.747380+0000 2024-04-25T09:29:41.269047+0000 1 periodic scrub scheduled @ 2024-04-26T15:51:25.232313+0000 2024-04-25T09:30:40.211 INFO:tasks.workunit.client.0.smithi134.stdout: 2024-04-25T09:30:40.211 INFO:tasks.workunit.client.0.smithi134.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/en/latest/dev/placement-group/#omap-statistics for further details. 2024-04-25T09:30:40.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: grep ' active.clean ' 2024-04-25T09:30:40.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:85: TEST_a_merge_empty: ceph pg ls 2024-04-25T09:30:40.640 INFO:tasks.workunit.client.0.smithi134.stdout:1.0 11 0 0 0 132748 0 0 0 11 active+clean 10s 24'7 45:95 [1,0,2]p1 [1,0,2]p1 2024-04-25T09:30:29.747380+0000 2024-04-25T09:29:41.269047+0000 1 periodic scrub scheduled @ 2024-04-26T15:51:25.232313+0000 2024-04-25T09:30:40.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:30:40.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:30:40.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:30:40.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:30:40.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:30:40.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:30:40.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:30:40.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:30:40.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:30:40.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:30:40.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:30:40.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:30:40.953 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:30:40.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:30:40.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:30:40.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:30:40.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:30:40.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:30:40.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:30:40.956 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:30:40.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:30:40.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:30:40.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:30:40.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:30:40.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:40.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:30:40.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451985 2024-04-25T09:30:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:30:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:30:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:30:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:30:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:30:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:30:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:30:40.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:30:40.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:30:40.987 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:30:40.987 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:30:40.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:30:40.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:30:40.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:30:40.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:30:40.989 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:30:40.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:30:40.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:30:40.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:30:40.991 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:30:40.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:30:40.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:30:40.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:30:40.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:30:40.993 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:30:40.994 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:30:40.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:30:40.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:30:40.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:30:40.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:40.997 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:30:40.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451985 2024-04-25T09:30:40.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:30:40.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:30:40.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:30:41.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:30:41.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:41.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:30:41.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.451985 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_merge_and_gap td/pg-split-merge 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:89: TEST_import_after_merge_and_gap: local dir=td/pg-split-merge 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:91: TEST_import_after_merge_and_gap: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:30:41.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:30:41.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:30:41.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:30:41.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:30:41.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:30:41.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:30:41.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:30:41.517 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:30:41.517 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:41.517 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:30:41.517 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:30:41.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:30:41.779 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:30:41.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:30:41.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:30:41.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:30:41.781 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:30:41.782 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:30:41.782 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:30:41.782 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:30:41.782 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:30:41.783 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:30:41.783 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:41.783 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:30:41.784 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451985/ceph-mon.a.asok 2024-04-25T09:30:41.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:30:41.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451985/ceph-mon.a.asok config get fsid 2024-04-25T09:30:41.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:30:41.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:30:41.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:30:41.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:30:41.900 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:30:41.900 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:30:41.900 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:30:41.900 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:30:41.901 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:30:41.901 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:41.901 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:30:41.901 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451985/ceph-mon.a.asok 2024-04-25T09:30:41.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:30:41.901 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451985/ceph-mon.a.asok config get mon_host 2024-04-25T09:30:42.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:92: TEST_import_after_merge_and_gap: run_mgr td/pg-split-merge x 2024-04-25T09:30:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:30:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:30:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:30:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:30:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:30:42.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:30:42.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:30:42.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:30:42.919 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:30:42.919 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:30:42.919 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:42.919 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:30:42.920 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:30:42.920 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:30:42.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:93: TEST_import_after_merge_and_gap: run_osd td/pg-split-merge 0 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:30:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:30:42.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:30:42.961 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:30:42.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c787144a-5beb-4391-bbea-edfe5c80e442 2024-04-25T09:30:42.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c787144a-5beb-4391-bbea-edfe5c80e442' 2024-04-25T09:30:42.962 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 c787144a-5beb-4391-bbea-edfe5c80e442 2024-04-25T09:30:42.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:30:42.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDCIipmyvdZOxAAbhixZCWzgvEFuGaqcvy5Ug== 2024-04-25T09:30:42.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDCIipmyvdZOxAAbhixZCWzgvEFuGaqcvy5Ug=="}' 2024-04-25T09:30:42.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c787144a-5beb-4391-bbea-edfe5c80e442 -i td/pg-split-merge/0/new.json 2024-04-25T09:30:43.331 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:30:43.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:30:43.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCIipmyvdZOxAAbhixZCWzgvEFuGaqcvy5Ug== --osd-uuid c787144a-5beb-4391-bbea-edfe5c80e442 2024-04-25T09:30:43.380 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:43.392+0000 7f78a94287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:43.408 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:43.424+0000 7f78a94287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:43.410 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:43.424+0000 7f78a94287c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:43.410 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:43.424+0000 7f78a94287c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:30:43.410 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:43.424+0000 7f78a94287c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:30:43.411 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:43.424+0000 7f78a94287c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:30:43.440 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:43.452+0000 7f78a94287c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:30:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:30:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:30:49.774 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:30:49.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:30:49.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:30:50.334 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:30:50.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:30:50.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:30:50.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:30:50.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:30:50.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:30:50.368 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:50.380+0000 7f2e97b807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:50.370 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:50.384+0000 7f2e97b807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:50.371 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:50.384+0000 7f2e97b807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:30:50.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:30:50.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:30:50.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:30:50.789 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:30:50.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:30:50.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:30:50.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:50.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:30:50.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:50.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:50.918 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:50.932+0000 7f2e97b807c0 -1 Falling back to public interface 2024-04-25T09:30:51.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:52.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:52.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:52.218 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:30:52.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:30:52.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:52.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:52.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:53.569 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:30:53.584+0000 7f2e97b807c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:30:53.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:53.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:53.662 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:30:53.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:30:53.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:53.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:54.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:55.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:55.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:55.192 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:30:55.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:30:55.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:55.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:55.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:56.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:56.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:56.650 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:30:56.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:30:56.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:56.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:57.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:58.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:58.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:58.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:30:58.095 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:30:58.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:58.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:30:58.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:30:59.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:30:59.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:30:59.657 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:30:59.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:30:59.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:30:59.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:00.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:01.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:01.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:01.078 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:31:01.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:31:01.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:01.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:01.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:02.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:02.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:02.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:31:02.683 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:31:02.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:02.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:03.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:04.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:04.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:04.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:31:04.137 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:31:04.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:04.138 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:04.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:05.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:05.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:05.682 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:31:05.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:31:05.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:05.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:06.104 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:07.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:07.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:07.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:31:07.106 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:31:07.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:07.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:07.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:08.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:08.896 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:08.896 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:31:08.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:31:08.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:08.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:10.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:11.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:11.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:11.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:31:11.091 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:31:11.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:11.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:11.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:12.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:12.894 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:12.894 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:31:12.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:31:12.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:12.895 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:13.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:14.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:14.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:14.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:31:14.729 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:31:14.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:14.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:15.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:16.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:16.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:16.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:31:16.534 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:31:16.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:16.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:17.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:18.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:18.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:18.435 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:31:18.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:31:18.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:18.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:18.880 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3063228234,v1:127.0.0.1:6803/3063228234] [v2:127.0.0.1:6804/3063228234,v1:127.0.0.1:6805/3063228234] exists,up c787144a-5beb-4391-bbea-edfe5c80e442 2024-04-25T09:31:18.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:31:18.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:31:18.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:31:18.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:95: TEST_import_after_merge_and_gap: ceph osd pool create foo 2 2024-04-25T09:31:19.459 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T09:31:19.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:96: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:31:19.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:31:19.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:31:19.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:31:19.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:31:19.476 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:31:19.476 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:31:19.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:31:19.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:31:19.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:31:19.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:31:19.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:31:19.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:31:19.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:31:19.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:31:19.575 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:31:20.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:31:20.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:31:20.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:31:20.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:31:20.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836489 2024-04-25T09:31:20.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836489 2024-04-25T09:31:20.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836489' 2024-04-25T09:31:20.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:31:20.220 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836489 2024-04-25T09:31:20.220 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:31:20.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:31:20.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836489 2024-04-25T09:31:20.221 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:31:20.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836489 2024-04-25T09:31:20.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-04-25T09:31:20.223 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836489 2024-04-25T09:31:20.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:31:20.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836487 -lt 21474836489 2024-04-25T09:31:20.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:31:21.648 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:31:21.648 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:31:22.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836489 -lt 21474836489 2024-04-25T09:31:22.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:31:22.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:31:22.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:31:22.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:31:22.623 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:31:22.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:31:22.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:31:22.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:31:22.623 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:31:22.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:31:22.624 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:31:23.054 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:31:23.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:31:23.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:31:23.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:31:23.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:31:23.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:31:23.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:31:23.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:97: TEST_import_after_merge_and_gap: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-04-25T09:31:23.612 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T09:31:23.612 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-04-25T09:31:23.612 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_462957 2024-04-25T09:31:23.613 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:31:23.613 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:31:24.612 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 98 82 0.080073 0.0800781 0.117569 0.178227 2024-04-25T09:31:25.612 INFO:tasks.workunit.client.0.smithi134.stdout: 2 16 224 208 0.101552 0.123047 0.142397 0.150164 2024-04-25T09:31:26.613 INFO:tasks.workunit.client.0.smithi134.stdout: 3 16 336 320 0.104156 0.109375 0.133701 0.149557 2024-04-25T09:31:27.613 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 3.1252 2024-04-25T09:31:27.613 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 336 2024-04-25T09:31:27.613 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 1024 2024-04-25T09:31:27.613 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 1024 2024-04-25T09:31:27.613 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 0.104993 2024-04-25T09:31:27.613 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 0.0219528 2024-04-25T09:31:27.613 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 0.123047 2024-04-25T09:31:27.614 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 0.0800781 2024-04-25T09:31:27.614 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 107 2024-04-25T09:31:27.614 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 22.4796 2024-04-25T09:31:27.614 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 126 2024-04-25T09:31:27.614 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 82 2024-04-25T09:31:27.614 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 0.148413 2024-04-25T09:31:27.614 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.0355853 2024-04-25T09:31:27.614 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 0.249566 2024-04-25T09:31:27.614 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.115653 2024-04-25T09:31:27.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:99: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:31:27.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:31:27.709 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:31:27.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:31:27.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:31:27.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:31:29.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:31:29.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:100: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-04-25T09:31:29.164 INFO:tasks.workunit.client.0.smithi134.stderr:Exporting 1.1 info 1.1( v 9'170 (0'0,9'170] local-lis/les=7/8 n=170 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:31:29.168 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:80f5f423:::benchmark_data_smithi134_462957_object61:head# 2024-04-25T09:31:29.169 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8136ddfc:::benchmark_data_smithi134_462957_object181:head# 2024-04-25T09:31:29.171 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:822e7d0d:::benchmark_data_smithi134_462957_object36:head# 2024-04-25T09:31:29.171 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:82344c27:::benchmark_data_smithi134_462957_object239:head# 2024-04-25T09:31:29.172 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:82ab8645:::benchmark_data_smithi134_462957_object312:head# 2024-04-25T09:31:29.174 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:83241f30:::benchmark_data_smithi134_462957_object186:head# 2024-04-25T09:31:29.174 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:833e47e2:::benchmark_data_smithi134_462957_object201:head# 2024-04-25T09:31:29.175 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:83909346:::benchmark_data_smithi134_462957_object165:head# 2024-04-25T09:31:29.176 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8486115e:::benchmark_data_smithi134_462957_object134:head# 2024-04-25T09:31:29.176 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:85d49e01:::benchmark_data_smithi134_462957_object114:head# 2024-04-25T09:31:29.176 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:86e69dbb:::benchmark_data_smithi134_462957_object334:head# 2024-04-25T09:31:29.177 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:87789c1f:::benchmark_data_smithi134_462957_object299:head# 2024-04-25T09:31:29.177 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:87e9176e:::benchmark_data_smithi134_462957_object176:head# 2024-04-25T09:31:29.178 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:88b81233:::benchmark_data_smithi134_462957_object197:head# 2024-04-25T09:31:29.178 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8a55344d:::benchmark_data_smithi134_462957_object163:head# 2024-04-25T09:31:29.178 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8b35579f:::benchmark_data_smithi134_462957_object101:head# 2024-04-25T09:31:29.179 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8c42d79e:::benchmark_data_smithi134_462957_object285:head# 2024-04-25T09:31:29.179 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8c5cb74b:::benchmark_data_smithi134_462957_object30:head# 2024-04-25T09:31:29.184 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8c777440:::benchmark_data_smithi134_462957_object69:head# 2024-04-25T09:31:29.185 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8d18392e:::benchmark_data_smithi134_462957_object42:head# 2024-04-25T09:31:29.185 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8d4a51df:::benchmark_data_smithi134_462957_object202:head# 2024-04-25T09:31:29.188 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8f9fc238:::benchmark_data_smithi134_462957_object222:head# 2024-04-25T09:31:29.189 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8fcd40aa:::benchmark_data_smithi134_462957_object13:head# 2024-04-25T09:31:29.190 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8ffd53b9:::benchmark_data_smithi134_462957_object152:head# 2024-04-25T09:31:29.190 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:912ceabf:::benchmark_data_smithi134_462957_object291:head# 2024-04-25T09:31:29.198 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:913e15cc:::benchmark_data_smithi134_462957_object283:head# 2024-04-25T09:31:29.199 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:92bf16d1:::benchmark_data_smithi134_462957_object195:head# 2024-04-25T09:31:29.200 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:92ed1f0f:::benchmark_data_smithi134_462957_object65:head# 2024-04-25T09:31:29.200 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9417108d:::benchmark_data_smithi134_462957_object261:head# 2024-04-25T09:31:29.201 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:954cc32c:::benchmark_data_smithi134_462957_object139:head# 2024-04-25T09:31:29.202 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9632b147:::benchmark_data_smithi134_462957_object154:head# 2024-04-25T09:31:29.202 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9726b3f5:::benchmark_data_smithi134_462957_object135:head# 2024-04-25T09:31:29.203 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:97aa80c9:::benchmark_data_smithi134_462957_object210:head# 2024-04-25T09:31:29.204 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9870b61d:::benchmark_data_smithi134_462957_object50:head# 2024-04-25T09:31:29.204 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9a3c902a:::benchmark_data_smithi134_462957_object66:head# 2024-04-25T09:31:29.204 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9a63a41b:::benchmark_data_smithi134_462957_object59:head# 2024-04-25T09:31:29.205 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9af4c21c:::benchmark_data_smithi134_462957_object99:head# 2024-04-25T09:31:29.205 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9c02c888:::benchmark_data_smithi134_462957_object100:head# 2024-04-25T09:31:29.205 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9d740ac7:::benchmark_data_smithi134_462957_object156:head# 2024-04-25T09:31:29.206 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9e32be9a:::benchmark_data_smithi134_462957_object54:head# 2024-04-25T09:31:29.206 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9e6056cd:::benchmark_data_smithi134_462957_object164:head# 2024-04-25T09:31:29.207 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9ebf06db:::benchmark_data_smithi134_462957_object284:head# 2024-04-25T09:31:29.207 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a045e8d8:::benchmark_data_smithi134_462957_object234:head# 2024-04-25T09:31:29.208 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a08b0a4d:::benchmark_data_smithi134_462957_object241:head# 2024-04-25T09:31:29.208 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a0c022f8:::benchmark_data_smithi134_462957_object226:head# 2024-04-25T09:31:29.209 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a1658627:::benchmark_data_smithi134_462957_object327:head# 2024-04-25T09:31:29.209 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a1bb4fbf:::benchmark_data_smithi134_462957_object198:head# 2024-04-25T09:31:29.209 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a2b4a5bd:::benchmark_data_smithi134_462957_object27:head# 2024-04-25T09:31:29.210 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a5ca7c00:::benchmark_data_smithi134_462957_object72:head# 2024-04-25T09:31:29.211 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a5f62c63:::benchmark_data_smithi134_462957_object316:head# 2024-04-25T09:31:29.212 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a63f834f:::benchmark_data_smithi134_462957_object157:head# 2024-04-25T09:31:29.212 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a715009b:::benchmark_data_smithi134_462957_object169:head# 2024-04-25T09:31:29.213 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a85dbb56:::benchmark_data_smithi134_462957_object307:head# 2024-04-25T09:31:29.214 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a8d1cd5f:::benchmark_data_smithi134_462957_object107:head# 2024-04-25T09:31:29.214 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a9621e84:::benchmark_data_smithi134_462957_object303:head# 2024-04-25T09:31:29.215 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a9a511c7:::benchmark_data_smithi134_462957_object321:head# 2024-04-25T09:31:29.215 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ab6edda1:::benchmark_data_smithi134_462957_object212:head# 2024-04-25T09:31:29.216 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:aba338b0:::benchmark_data_smithi134_462957_object184:head# 2024-04-25T09:31:29.217 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:abdd83aa:::benchmark_data_smithi134_462957_object85:head# 2024-04-25T09:31:29.217 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ac85576b:::benchmark_data_smithi134_462957_object282:head# 2024-04-25T09:31:29.218 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ae25ce97:::benchmark_data_smithi134_462957_object75:head# 2024-04-25T09:31:29.218 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b1f8b3b5:::benchmark_data_smithi134_462957_object160:head# 2024-04-25T09:31:29.219 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b249419b:::benchmark_data_smithi134_462957_object275:head# 2024-04-25T09:31:29.219 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b289940b:::benchmark_data_smithi134_462957_object26:head# 2024-04-25T09:31:29.220 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b29d1223:::benchmark_data_smithi134_462957_object167:head# 2024-04-25T09:31:29.221 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b4ae5688:::benchmark_data_smithi134_462957_object182:head# 2024-04-25T09:31:29.222 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b5565581:::benchmark_data_smithi134_462957_object22:head# 2024-04-25T09:31:29.222 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b56627a7:::benchmark_data_smithi134_462957_object108:head# 2024-04-25T09:31:29.223 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b6ff86cd:::benchmark_data_smithi134_462957_object166:head# 2024-04-25T09:31:29.224 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b7edf419:::benchmark_data_smithi134_462957_object84:head# 2024-04-25T09:31:29.224 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:31:29.225 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b8f154ff:::benchmark_data_smithi134_462957_object206:head# 2024-04-25T09:31:29.226 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b93b7738:::benchmark_data_smithi134_462957_object92:head# 2024-04-25T09:31:29.226 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b997874c:::benchmark_data_smithi134_462957_object320:head# 2024-04-25T09:31:29.226 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ba34c80e:::benchmark_data_smithi134_462957_object29:head# 2024-04-25T09:31:29.227 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bb51ab09:::benchmark_data_smithi134_462957_object246:head# 2024-04-25T09:31:29.228 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bb9ed292:::benchmark_data_smithi134_462957_object44:head# 2024-04-25T09:31:29.228 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bc0fe2eb:::benchmark_data_smithi134_462957_object105:head# 2024-04-25T09:31:29.229 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bd33c13c:::benchmark_data_smithi134_462957_object121:head# 2024-04-25T09:31:29.229 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bd52ec16:::benchmark_data_smithi134_462957_object68:head# 2024-04-25T09:31:29.230 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:be19e778:::benchmark_data_smithi134_462957_object238:head# 2024-04-25T09:31:29.231 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bf64a100:::benchmark_data_smithi134_462957_object83:head# 2024-04-25T09:31:29.231 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c03f464f:::benchmark_data_smithi134_462957_object18:head# 2024-04-25T09:31:29.232 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c0f283ce:::benchmark_data_smithi134_462957_object124:head# 2024-04-25T09:31:29.232 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c16c7b78:::benchmark_data_smithi134_462957_object2:head# 2024-04-25T09:31:29.233 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c18f6438:::benchmark_data_smithi134_462957_object220:head# 2024-04-25T09:31:29.233 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c244846a:::benchmark_data_smithi134_462957_object329:head# 2024-04-25T09:31:29.234 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c274bcc9:::benchmark_data_smithi134_462957_object332:head# 2024-04-25T09:31:29.235 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c2e9f6b4:::benchmark_data_smithi134_462957_object330:head# 2024-04-25T09:31:29.235 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c35cfddf:::benchmark_data_smithi134_462957_object200:head# 2024-04-25T09:31:29.236 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c47b78d0:::benchmark_data_smithi134_462957_object117:head# 2024-04-25T09:31:29.237 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c70a7496:::benchmark_data_smithi134_462957_object60:head# 2024-04-25T09:31:29.238 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c73308dd:::benchmark_data_smithi134_462957_object258:head# 2024-04-25T09:31:29.238 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c74b145d:::benchmark_data_smithi134_462957_object146:head# 2024-04-25T09:31:29.239 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c7d37ef1:::benchmark_data_smithi134_462957_object125:head# 2024-04-25T09:31:29.240 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c84d6c63:::benchmark_data_smithi134_462957_object221:head# 2024-04-25T09:31:29.240 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c8d637f7:::benchmark_data_smithi134_462957_object306:head# 2024-04-25T09:31:29.241 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c9e347b1:::benchmark_data_smithi134_462957_object242:head# 2024-04-25T09:31:29.241 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ca46b863:::benchmark_data_smithi134_462957_object57:head# 2024-04-25T09:31:29.242 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ca795a74:::benchmark_data_smithi134_462957_object81:head# 2024-04-25T09:31:29.243 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cc49d25d:::benchmark_data_smithi134_462957_object144:head# 2024-04-25T09:31:29.244 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ccd8cbfe:::benchmark_data_smithi134_462957_object5:head# 2024-04-25T09:31:29.245 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cce05002:::benchmark_data_smithi134_462957_object130:head# 2024-04-25T09:31:29.245 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cd79c5f8:::benchmark_data_smithi134_462957_object76:head# 2024-04-25T09:31:29.246 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cdb281d5:::benchmark_data_smithi134_462957_object46:head# 2024-04-25T09:31:29.247 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cdc69e54:::benchmark_data_smithi134_462957_object252:head# 2024-04-25T09:31:29.247 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cf2a4062:::benchmark_data_smithi134_462957_object199:head# 2024-04-25T09:31:29.248 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cf35812c:::benchmark_data_smithi134_462957_object243:head# 2024-04-25T09:31:29.249 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cf693ea1:::benchmark_data_smithi134_462957_object147:head# 2024-04-25T09:31:29.250 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d0dad50c:::benchmark_data_smithi134_462957_object97:head# 2024-04-25T09:31:29.250 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d1761cb1:::benchmark_data_smithi134_462957_object251:head# 2024-04-25T09:31:29.251 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d1b7fc50:::benchmark_data_smithi134_462957_object211:head# 2024-04-25T09:31:29.251 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d2bd3105:::benchmark_data_smithi134_462957_object277:head# 2024-04-25T09:31:29.252 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d3201ed2:::benchmark_data_smithi134_462957_object45:head# 2024-04-25T09:31:29.253 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d423debd:::benchmark_data_smithi134_462957_object115:head# 2024-04-25T09:31:29.254 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d4c4f287:::benchmark_data_smithi134_462957_object21:head# 2024-04-25T09:31:29.254 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d50a78a1:::benchmark_data_smithi134_462957_object25:head# 2024-04-25T09:31:29.254 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d5af3bd1:::benchmark_data_smithi134_462957_object180:head# 2024-04-25T09:31:29.255 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d5ef9c20:::benchmark_data_smithi134_462957_object70:head# 2024-04-25T09:31:29.256 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d65dc781:::benchmark_data_smithi134_462957_object323:head# 2024-04-25T09:31:29.257 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d6872c2f:::benchmark_data_smithi134_462957_object192:head# 2024-04-25T09:31:29.257 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d6cbb661:::benchmark_data_smithi134_462957_object279:head# 2024-04-25T09:31:29.258 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d6d8a7de:::benchmark_data_smithi134_462957_object175:head# 2024-04-25T09:31:29.259 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d9df41a0:::benchmark_data_smithi134_462957_object288:head# 2024-04-25T09:31:29.259 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dba2aa95:::benchmark_data_smithi134_462957_object31:head# 2024-04-25T09:31:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dc540d1b:::benchmark_data_smithi134_462957_object189:head# 2024-04-25T09:31:29.261 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dc6e92dd:::benchmark_data_smithi134_462957_object159:head# 2024-04-25T09:31:29.262 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dcca6922:::benchmark_data_smithi134_462957_object311:head# 2024-04-25T09:31:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd1204a0:::benchmark_data_smithi134_462957_object71:head# 2024-04-25T09:31:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd3e6d82:::benchmark_data_smithi134_462957_object262:head# 2024-04-25T09:31:29.264 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dfc1c8cf:::benchmark_data_smithi134_462957_object229:head# 2024-04-25T09:31:29.265 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e06b8840:::benchmark_data_smithi134_462957_object91:head# 2024-04-25T09:31:29.265 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e06cab0a:::benchmark_data_smithi134_462957_object23:head# 2024-04-25T09:31:29.266 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e28f6f75:::benchmark_data_smithi134_462957_object300:head# 2024-04-25T09:31:29.267 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e2be0804:::benchmark_data_smithi134_462957_object235:head# 2024-04-25T09:31:29.267 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e2c8c0bf:::benchmark_data_smithi134_462957_object40:head# 2024-04-25T09:31:29.268 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e3a40c39:::benchmark_data_smithi134_462957_object63:head# 2024-04-25T09:31:29.268 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e5228945:::benchmark_data_smithi134_462957_object293:head# 2024-04-25T09:31:29.269 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e614399c:::benchmark_data_smithi134_462957_object110:head# 2024-04-25T09:31:29.270 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e65f056d:::benchmark_data_smithi134_462957_object196:head# 2024-04-25T09:31:29.270 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e6a6c722:::benchmark_data_smithi134_462957_object264:head# 2024-04-25T09:31:29.271 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e74bd302:::benchmark_data_smithi134_462957_object131:head# 2024-04-25T09:31:29.272 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e7aa26a1:::benchmark_data_smithi134_462957_object129:head# 2024-04-25T09:31:29.272 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e8378195:::benchmark_data_smithi134_462957_object256:head# 2024-04-25T09:31:29.273 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e8ce525c:::benchmark_data_smithi134_462957_object155:head# 2024-04-25T09:31:29.273 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e8f5abd5:::benchmark_data_smithi134_462957_object56:head# 2024-04-25T09:31:29.275 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e985cf7e:::benchmark_data_smithi134_462957_object287:head# 2024-04-25T09:31:29.275 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e9f08f42:::benchmark_data_smithi134_462957_object187:head# 2024-04-25T09:31:29.276 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ea484b6f:::benchmark_data_smithi134_462957_object82:head# 2024-04-25T09:31:29.277 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ee73ec6d:::benchmark_data_smithi134_462957_object33:head# 2024-04-25T09:31:29.277 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f01022da:::benchmark_data_smithi134_462957_object324:head# 2024-04-25T09:31:29.278 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f0800d50:::benchmark_data_smithi134_462957_object217:head# 2024-04-25T09:31:29.279 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f0f00dfc:::benchmark_data_smithi134_462957_object185:head# 2024-04-25T09:31:29.279 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f0f2cea2:::benchmark_data_smithi134_462957_object317:head# 2024-04-25T09:31:29.280 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f131ea49:::benchmark_data_smithi134_462957_object15:head# 2024-04-25T09:31:29.281 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f3c72ea5:::benchmark_data_smithi134_462957_object173:head# 2024-04-25T09:31:29.282 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f581c1b4:::benchmark_data_smithi134_462957_object218:head# 2024-04-25T09:31:29.282 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f63f8b4d:::benchmark_data_smithi134_462957_object247:head# 2024-04-25T09:31:29.283 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f70159a7:::benchmark_data_smithi134_462957_object208:head# 2024-04-25T09:31:29.284 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f741ec8b:::benchmark_data_smithi134_462957_object325:head# 2024-04-25T09:31:29.284 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f7f2b0bd:::benchmark_data_smithi134_462957_object302:head# 2024-04-25T09:31:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f8d2618b:::benchmark_data_smithi134_462957_object255:head# 2024-04-25T09:31:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f8f97fba:::benchmark_data_smithi134_462957_object178:head# 2024-04-25T09:31:29.286 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fa5b5d21:::benchmark_data_smithi134_462957_object274:head# 2024-04-25T09:31:29.287 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:faa372e4:::benchmark_data_smithi134_462957_object228:head# 2024-04-25T09:31:29.287 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fb20bb8f:::benchmark_data_smithi134_462957_object190:head# 2024-04-25T09:31:29.288 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fc414211:::benchmark_data_smithi134_462957_object79:head# 2024-04-25T09:31:29.289 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fc7f33f8:::benchmark_data_smithi134_462957_object37:head# 2024-04-25T09:31:29.290 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fee7f4ff:::benchmark_data_smithi134_462957_object204:head# 2024-04-25T09:31:29.290 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ff13f1d8:::benchmark_data_smithi134_462957_object49:head# 2024-04-25T09:31:29.291 INFO:tasks.workunit.client.0.smithi134.stderr:Export successful 2024-04-25T09:31:29.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:101: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:31:29.717 INFO:tasks.workunit.client.0.smithi134.stderr:Exporting 1.0 info 1.0( v 9'167 (0'0,9'167] local-lis/les=7/8 n=167 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:31:29.719 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:014fbc01:::benchmark_data_smithi134_462957_object116:head# 2024-04-25T09:31:29.720 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:01ae4a14:::benchmark_data_smithi134_462957_object335:head# 2024-04-25T09:31:29.720 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:022eaaef:::benchmark_data_smithi134_462957_object132:head# 2024-04-25T09:31:29.720 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:04df5908:::benchmark_data_smithi134_462957_object10:head# 2024-04-25T09:31:29.721 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:05e918e6:::benchmark_data_smithi134_462957_object87:head# 2024-04-25T09:31:29.721 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:08057fd0:::benchmark_data_smithi134_462957_object95:head# 2024-04-25T09:31:29.721 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:08313bb1:::benchmark_data_smithi134_462957_object240:head# 2024-04-25T09:31:29.721 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0958de82:::benchmark_data_smithi134_462957_object305:head# 2024-04-25T09:31:29.722 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:09655936:::benchmark_data_smithi134_462957_object278:head# 2024-04-25T09:31:29.723 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0a7e421b:::benchmark_data_smithi134_462957_object39:head# 2024-04-25T09:31:29.723 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0be5f62e:::benchmark_data_smithi134_462957_object126:head# 2024-04-25T09:31:29.723 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0ce6c2ff:::benchmark_data_smithi134_462957_object48:head# 2024-04-25T09:31:29.724 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0d00a542:::benchmark_data_smithi134_462957_object112:head# 2024-04-25T09:31:29.724 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0d256506:::benchmark_data_smithi134_462957_object177:head# 2024-04-25T09:31:29.725 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0e18ad9b:::benchmark_data_smithi134_462957_object52:head# 2024-04-25T09:31:29.725 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0f50edf7:::benchmark_data_smithi134_462957_object304:head# 2024-04-25T09:31:29.726 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0f680139:::benchmark_data_smithi134_462957_object310:head# 2024-04-25T09:31:29.726 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0fc29821:::benchmark_data_smithi134_462957_object276:head# 2024-04-25T09:31:29.726 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:109d776b:::benchmark_data_smithi134_462957_object331:head# 2024-04-25T09:31:29.726 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1131e4a2:::benchmark_data_smithi134_462957_object266:head# 2024-04-25T09:31:29.727 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:11a2460d:::benchmark_data_smithi134_462957_object267:head# 2024-04-25T09:31:29.728 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1255ae71:::benchmark_data_smithi134_462957_object123:head# 2024-04-25T09:31:29.728 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:134c0c2c:::benchmark_data_smithi134_462957_object140:head# 2024-04-25T09:31:29.728 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:136d3c79:::benchmark_data_smithi134_462957_object216:head# 2024-04-25T09:31:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:14c28473:::benchmark_data_smithi134_462957_object143:head# 2024-04-25T09:31:29.729 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:14d46dea:::benchmark_data_smithi134_462957_object322:head# 2024-04-25T09:31:29.730 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:151932bc:::benchmark_data_smithi134_462957_object127:head# 2024-04-25T09:31:29.730 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:15788a9d:::benchmark_data_smithi134_462957_object215:head# 2024-04-25T09:31:29.731 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1607bd5a:::benchmark_data_smithi134_462957_object24:head# 2024-04-25T09:31:29.731 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:16357f0a:::benchmark_data_smithi134_462957_object16:head# 2024-04-25T09:31:29.731 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:16baa595:::benchmark_data_smithi134_462957_object309:head# 2024-04-25T09:31:29.732 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1965afdf:::benchmark_data_smithi134_462957_object89:head# 2024-04-25T09:31:29.732 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1affb4ca:::benchmark_data_smithi134_462957_object62:head# 2024-04-25T09:31:29.732 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1b230e12:::benchmark_data_smithi134_462957_object118:head# 2024-04-25T09:31:29.733 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1bd04fc7:::benchmark_data_smithi134_462957_object188:head# 2024-04-25T09:31:29.733 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1d14325e:::benchmark_data_smithi134_462957_object41:head# 2024-04-25T09:31:29.734 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1d8f10c1:::benchmark_data_smithi134_462957_object86:head# 2024-04-25T09:31:29.734 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1da2d015:::benchmark_data_smithi134_462957_object64:head# 2024-04-25T09:31:29.734 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1dad9b9c:::benchmark_data_smithi134_462957_object257:head# 2024-04-25T09:31:29.735 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1fb02fa0:::benchmark_data_smithi134_462957_object193:head# 2024-04-25T09:31:29.735 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:203c2195:::benchmark_data_smithi134_462957_object254:head# 2024-04-25T09:31:29.736 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:22302b8d:::benchmark_data_smithi134_462957_object80:head# 2024-04-25T09:31:29.736 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:228f6133:::benchmark_data_smithi134_462957_object74:head# 2024-04-25T09:31:29.737 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:23e9d7b5:::benchmark_data_smithi134_462957_object245:head# 2024-04-25T09:31:29.737 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2426c803:::benchmark_data_smithi134_462957_object290:head# 2024-04-25T09:31:29.738 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:244ef85c:::benchmark_data_smithi134_462957_object170:head# 2024-04-25T09:31:29.738 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2513bbe2:::benchmark_data_smithi134_462957_object207:head# 2024-04-25T09:31:29.739 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2568f8b5:::benchmark_data_smithi134_462957_object8:head# 2024-04-25T09:31:29.739 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:260f1533:::benchmark_data_smithi134_462957_object191:head# 2024-04-25T09:31:29.740 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:27170e69:::benchmark_data_smithi134_462957_object34:head# 2024-04-25T09:31:29.740 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:275fd7c3:::benchmark_data_smithi134_462957_object225:head# 2024-04-25T09:31:29.741 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:28ef7a99:::benchmark_data_smithi134_462957_object162:head# 2024-04-25T09:31:29.741 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2b3177c6:::benchmark_data_smithi134_462957_object298:head# 2024-04-25T09:31:29.742 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2b9356c2:::benchmark_data_smithi134_462957_object270:head# 2024-04-25T09:31:29.742 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2cb634ef:::benchmark_data_smithi134_462957_object297:head# 2024-04-25T09:31:29.743 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2ccb43ac:::benchmark_data_smithi134_462957_object28:head# 2024-04-25T09:31:29.743 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2cf40efb:::benchmark_data_smithi134_462957_object289:head# 2024-04-25T09:31:29.743 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2d9f151f:::benchmark_data_smithi134_462957_object73:head# 2024-04-25T09:31:29.744 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2dfb304e:::benchmark_data_smithi134_462957_object138:head# 2024-04-25T09:31:29.744 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2e76756e:::benchmark_data_smithi134_462957_object174:head# 2024-04-25T09:31:29.745 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2f2a62da:::benchmark_data_smithi134_462957_object230:head# 2024-04-25T09:31:29.745 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2f954ef6:::benchmark_data_smithi134_462957_object120:head# 2024-04-25T09:31:29.746 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:302de82f:::benchmark_data_smithi134_462957_object268:head# 2024-04-25T09:31:29.746 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:314aacf8:::benchmark_data_smithi134_462957_object98:head# 2024-04-25T09:31:29.746 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:328e347c:::benchmark_data_smithi134_462957_object203:head# 2024-04-25T09:31:29.747 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:33056cad:::benchmark_data_smithi134_462957_object248:head# 2024-04-25T09:31:29.747 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3342be62:::benchmark_data_smithi134_462957_object158:head# 2024-04-25T09:31:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:33c19275:::benchmark_data_smithi134_462957_object43:head# 2024-04-25T09:31:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:34187219:::benchmark_data_smithi134_462957_object294:head# 2024-04-25T09:31:29.749 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:344b976e:::benchmark_data_smithi134_462957_object231:head# 2024-04-25T09:31:29.749 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:358811ac:::benchmark_data_smithi134_462957_object142:head# 2024-04-25T09:31:29.750 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:361e89fa:::benchmark_data_smithi134_462957_object308:head# 2024-04-25T09:31:29.751 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:369c019f:::benchmark_data_smithi134_462957_object313:head# 2024-04-25T09:31:29.751 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:39249c23:::benchmark_data_smithi134_462957_object161:head# 2024-04-25T09:31:29.751 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3a8ee0cf:::benchmark_data_smithi134_462957_object151:head# 2024-04-25T09:31:29.752 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3b9ee75a:::benchmark_data_smithi134_462957_object326:head# 2024-04-25T09:31:29.752 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3e5f4585:::benchmark_data_smithi134_462957_object47:head# 2024-04-25T09:31:29.753 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3ed5c4c3:::benchmark_data_smithi134_462957_object205:head# 2024-04-25T09:31:29.753 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3fc6ca95:::benchmark_data_smithi134_462957_object12:head# 2024-04-25T09:31:29.754 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:40470fd1:::benchmark_data_smithi134_462957_object17:head# 2024-04-25T09:31:29.754 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:417961ce:::benchmark_data_smithi134_462957_object319:head# 2024-04-25T09:31:29.754 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:418ec0ee:::benchmark_data_smithi134_462957_object149:head# 2024-04-25T09:31:29.755 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:422d92f6:::benchmark_data_smithi134_462957_object272:head# 2024-04-25T09:31:29.755 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:435e2086:::benchmark_data_smithi134_462957_object171:head# 2024-04-25T09:31:29.756 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:444a49fd:::benchmark_data_smithi134_462957_object328:head# 2024-04-25T09:31:29.756 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:44a8dd41:::benchmark_data_smithi134_462957_object3:head# 2024-04-25T09:31:29.757 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:467046b5:::benchmark_data_smithi134_462957_object265:head# 2024-04-25T09:31:29.757 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4672b82d:::benchmark_data_smithi134_462957_object295:head# 2024-04-25T09:31:29.758 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:47196e71:::benchmark_data_smithi134_462957_object269:head# 2024-04-25T09:31:29.758 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:478e238f:::benchmark_data_smithi134_462957_object213:head# 2024-04-25T09:31:29.759 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:47e25b7f:::benchmark_data_smithi134_462957_object249:head# 2024-04-25T09:31:29.759 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:47f5dd1f:::benchmark_data_smithi134_462957_object122:head# 2024-04-25T09:31:29.760 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:49e180d1:::benchmark_data_smithi134_462957_object88:head# 2024-04-25T09:31:29.760 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4a676de3:::benchmark_data_smithi134_462957_object263:head# 2024-04-25T09:31:29.760 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4b79b182:::benchmark_data_smithi134_462957_object137:head# 2024-04-25T09:31:29.761 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4cb6f1b7:::benchmark_data_smithi134_462957_object78:head# 2024-04-25T09:31:29.761 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4e62c3c1:::benchmark_data_smithi134_462957_object53:head# 2024-04-25T09:31:29.762 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4e981d63:::benchmark_data_smithi134_462957_object227:head# 2024-04-25T09:31:29.762 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4fda476d:::benchmark_data_smithi134_462957_object194:head# 2024-04-25T09:31:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:504c6844:::benchmark_data_smithi134_462957_object94:head# 2024-04-25T09:31:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:504ee13c:::benchmark_data_smithi134_462957_object93:head# 2024-04-25T09:31:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:511d8705:::benchmark_data_smithi134_462957_object271:head# 2024-04-25T09:31:29.764 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:51b089df:::benchmark_data_smithi134_462957_object273:head# 2024-04-25T09:31:29.765 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5266c200:::benchmark_data_smithi134_462957_object32:head# 2024-04-25T09:31:29.765 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:542d36c1:::benchmark_data_smithi134_462957_object4:head# 2024-04-25T09:31:29.765 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:544671af:::benchmark_data_smithi134_462957_object172:head# 2024-04-25T09:31:29.766 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:549eba21:::benchmark_data_smithi134_462957_object133:head# 2024-04-25T09:31:29.766 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:556d8a79:::benchmark_data_smithi134_462957_object214:head# 2024-04-25T09:31:29.767 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:556e8b3c:::benchmark_data_smithi134_462957_object96:head# 2024-04-25T09:31:29.767 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:55950821:::benchmark_data_smithi134_462957_object141:head# 2024-04-25T09:31:29.767 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:575ccb6b:::benchmark_data_smithi134_462957_object280:head# 2024-04-25T09:31:29.768 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:57a020db:::benchmark_data_smithi134_462957_object286:head# 2024-04-25T09:31:29.768 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:58587487:::benchmark_data_smithi134_462957_object51:head# 2024-04-25T09:31:29.769 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5c373486:::benchmark_data_smithi134_462957_object14:head# 2024-04-25T09:31:29.769 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5dada283:::benchmark_data_smithi134_462957_object77:head# 2024-04-25T09:31:29.770 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5e48ae44:::benchmark_data_smithi134_462957_object333:head# 2024-04-25T09:31:29.770 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:60d506d0:::benchmark_data_smithi134_462957_object111:head# 2024-04-25T09:31:29.771 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:61342236:::benchmark_data_smithi134_462957_object301:head# 2024-04-25T09:31:29.771 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:61d1b2a4:::benchmark_data_smithi134_462957_object179:head# 2024-04-25T09:31:29.772 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:621d43c5:::benchmark_data_smithi134_462957_object102:head# 2024-04-25T09:31:29.772 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:62545d41:::benchmark_data_smithi134_462957_object318:head# 2024-04-25T09:31:29.773 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:632b2ff2:::benchmark_data_smithi134_462957_object113:head# 2024-04-25T09:31:29.773 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:63effe0b:::benchmark_data_smithi134_462957_object1:head# 2024-04-25T09:31:29.773 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:63f38e63:::benchmark_data_smithi134_462957_object314:head# 2024-04-25T09:31:29.774 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:645caf70:::benchmark_data_smithi134_462957_object9:head# 2024-04-25T09:31:29.775 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:64cb656a:::benchmark_data_smithi134_462957_object38:head# 2024-04-25T09:31:29.775 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:651d1c11:::benchmark_data_smithi134_462957_object219:head# 2024-04-25T09:31:29.776 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:65698cf1:::benchmark_data_smithi134_462957_object315:head# 2024-04-25T09:31:29.776 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:660b1875:::benchmark_data_smithi134_462957_object153:head# 2024-04-25T09:31:29.777 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:66e4bb20:::benchmark_data_smithi134_462957_object259:head# 2024-04-25T09:31:29.778 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:67406b53:::benchmark_data_smithi134_462957_object168:head# 2024-04-25T09:31:29.778 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6753f201:::benchmark_data_smithi134_462957_object19:head# 2024-04-25T09:31:29.779 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:67de4446:::benchmark_data_smithi134_462957_object11:head# 2024-04-25T09:31:29.779 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68431e33:::benchmark_data_smithi134_462957_object6:head# 2024-04-25T09:31:29.780 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68da1af8:::benchmark_data_smithi134_462957_object224:head# 2024-04-25T09:31:29.780 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68e29e1b:::benchmark_data_smithi134_462957_object223:head# 2024-04-25T09:31:29.781 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68ff3868:::benchmark_data_smithi134_462957_object104:head# 2024-04-25T09:31:29.781 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:69444b58:::benchmark_data_smithi134_462957_object236:head# 2024-04-25T09:31:29.782 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6bef445e:::benchmark_data_smithi134_462957_object145:head# 2024-04-25T09:31:29.782 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6c6b0a68:::benchmark_data_smithi134_462957_object106:head# 2024-04-25T09:31:29.783 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6cb36ab9:::benchmark_data_smithi134_462957_object128:head# 2024-04-25T09:31:29.783 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6cb830fe:::benchmark_data_smithi134_462957_object281:head# 2024-04-25T09:31:29.784 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6d038ea7:::benchmark_data_smithi134_462957_object250:head# 2024-04-25T09:31:29.784 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6dfad7ef:::benchmark_data_smithi134_462957_object253:head# 2024-04-25T09:31:29.785 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6e8445ed:::benchmark_data_smithi134_462957_object119:head# 2024-04-25T09:31:29.785 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6ea927e3:::benchmark_data_smithi134_462957_object67:head# 2024-04-25T09:31:29.786 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6edbe582:::benchmark_data_smithi134_462957_object260:head# 2024-04-25T09:31:29.786 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6f922ade:::benchmark_data_smithi134_462957_object136:head# 2024-04-25T09:31:29.787 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6fe808b0:::benchmark_data_smithi134_462957_object7:head# 2024-04-25T09:31:29.787 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:71c24ec6:::benchmark_data_smithi134_462957_object35:head# 2024-04-25T09:31:29.788 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7280df13:::benchmark_data_smithi134_462957_object109:head# 2024-04-25T09:31:29.788 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:72b74ac7:::benchmark_data_smithi134_462957_object292:head# 2024-04-25T09:31:29.789 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:72d020e0:::benchmark_data_smithi134_462957_object20:head# 2024-04-25T09:31:29.789 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:73570709:::benchmark_data_smithi134_462957_object244:head# 2024-04-25T09:31:29.790 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:73fa9d47:::benchmark_data_smithi134_462957_object0:head# 2024-04-25T09:31:29.791 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:76387380:::benchmark_data_smithi134_462957_object103:head# 2024-04-25T09:31:29.791 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:77a61cee:::benchmark_data_smithi134_462957_object237:head# 2024-04-25T09:31:29.791 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7826dd0b:::benchmark_data_smithi134_462957_object233:head# 2024-04-25T09:31:29.792 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7958a491:::benchmark_data_smithi134_462957_object232:head# 2024-04-25T09:31:29.793 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7a5eed6c:::benchmark_data_smithi134_462957_object55:head# 2024-04-25T09:31:29.793 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7bed12e2:::benchmark_data_smithi134_462957_object90:head# 2024-04-25T09:31:29.794 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7c53f004:::benchmark_data_smithi134_462957_object183:head# 2024-04-25T09:31:29.794 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7cc751e9:::benchmark_data_smithi134_462957_object150:head# 2024-04-25T09:31:29.795 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7de067cb:::benchmark_data_smithi134_462957_object148:head# 2024-04-25T09:31:29.795 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7e10a619:::benchmark_data_smithi134_462957_object296:head# 2024-04-25T09:31:29.796 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7f2f5709:::benchmark_data_smithi134_462957_object58:head# 2024-04-25T09:31:29.796 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7f4d9c89:::benchmark_data_smithi134_462957_object209:head# 2024-04-25T09:31:29.797 INFO:tasks.workunit.client.0.smithi134.stderr:Export successful 2024-04-25T09:31:30.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:102: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:31:30.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:31:30.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:31:30.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:31:30.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:31:30.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:31:30.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:31:30.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:31:30.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:31:30.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:31:30.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:31:30.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:31:30.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:31:30.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:31:30.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:31:30.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:31:30.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:31:30.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:31:30.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:31:30.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:31:30.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:31:30.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:31:30.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:31:30.092 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:31:30.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:31:30.093 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:31:30.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:31:30.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:31:30.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:31:30.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:31:30.127 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:31:30.139+0000 7fa65c9d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:30.129 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:31:30.143+0000 7fa65c9d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:30.131 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:31:30.143+0000 7fa65c9d97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:31:30.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:31:30.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:31:30.508 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:31:30.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:31:30.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:31:30.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:31:30.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:30.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:31:30.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:30.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:30.698 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:31:30.711+0000 7fa65c9d97c0 -1 Falling back to public interface 2024-04-25T09:31:30.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:31.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:31.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:31.998 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:31:31.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:31:31.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:31.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:32.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:33.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:33.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:33.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:31:33.427 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:31:33.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:33.427 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:33.840 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:31:33.851+0000 7fa65c9d97c0 -1 osd.0 9 log_to_monitors true 2024-04-25T09:31:33.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:31:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:31:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:31:34.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:31:34.865 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:31:34.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:31:34.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:31:35.294 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/3711942453,v1:127.0.0.1:6803/3711942453] [v2:127.0.0.1:6804/3711942453,v1:127.0.0.1:6805/3711942453] exists,up c787144a-5beb-4391-bbea-edfe5c80e442 2024-04-25T09:31:35.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:31:35.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:31:35.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:31:35.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:104: TEST_import_after_merge_and_gap: ceph osd pool set foo pg_num 1 2024-04-25T09:31:36.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:105: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:31:41.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: ceph daemon osd.0 perf dump 2024-04-25T09:31:41.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: jq .osd.numpg 2024-04-25T09:31:41.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:106: TEST_import_after_merge_and_gap: grep 2 2024-04-25T09:31:41.140 INFO:tasks.workunit.client.0.smithi134.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-25T09:31:41.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:107: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:31:41.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:31:41.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:31:41.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:31:41.143 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:31:41.143 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:31:41.144 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:31:41.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:31:41.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:31:41.144 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:31:41.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:31:41.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:31:41.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:31:41.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:31:41.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:31:41.236 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:31:41.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:31:41.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:31:41.647 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:31:41.647 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:31:41.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607556 2024-04-25T09:31:41.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607556 2024-04-25T09:31:41.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-51539607556' 2024-04-25T09:31:41.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:31:41.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-51539607556 2024-04-25T09:31:41.879 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:31:41.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:31:41.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-51539607556 2024-04-25T09:31:41.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:31:41.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607556 2024-04-25T09:31:41.883 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 51539607556 2024-04-25T09:31:41.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-04-25T09:31:41.883 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:31:42.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607554 -lt 51539607556 2024-04-25T09:31:42.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:31:43.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:31:43.345 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:31:43.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607556 -lt 51539607556 2024-04-25T09:31:43.784 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:31:43.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:31:43.785 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:31:44.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:31:44.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:31:44.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:31:44.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:31:44.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:31:44.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:31:44.339 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:31:44.340 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:31:44.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:31:44.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:31:44.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:31:44.797 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:31:45.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:31:45.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:31:45.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:31:45.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:110: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:31:45.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:31:45.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:31:45.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:31:45.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:31:45.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:31:45.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:31:45.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:111: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:31:46.918 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T09:31:46.919 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T09:31:46.919 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:31:47.544 INFO:tasks.workunit.client.0.smithi134.stdout:Remove successful 2024-04-25T09:31:47.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:113: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:31:48.900 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e20 pg_nums {1={7=2,19=1}} deleted_pools ) 2024-04-25T09:31:48.900 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.1 2024-04-25T09:31:48.901 INFO:tasks.workunit.client.0.smithi134.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:31:48.901 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:80f5f423:::benchmark_data_smithi134_462957_object61:head# 2024-04-25T09:31:48.901 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:48.972 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8136ddfc:::benchmark_data_smithi134_462957_object181:head# 2024-04-25T09:31:48.972 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.005 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:822e7d0d:::benchmark_data_smithi134_462957_object36:head# 2024-04-25T09:31:49.006 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.047 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:82344c27:::benchmark_data_smithi134_462957_object239:head# 2024-04-25T09:31:49.047 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.089 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:82ab8645:::benchmark_data_smithi134_462957_object312:head# 2024-04-25T09:31:49.089 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.131 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:83241f30:::benchmark_data_smithi134_462957_object186:head# 2024-04-25T09:31:49.131 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.181 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:833e47e2:::benchmark_data_smithi134_462957_object201:head# 2024-04-25T09:31:49.181 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.231 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:83909346:::benchmark_data_smithi134_462957_object165:head# 2024-04-25T09:31:49.231 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.273 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8486115e:::benchmark_data_smithi134_462957_object134:head# 2024-04-25T09:31:49.273 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.315 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:85d49e01:::benchmark_data_smithi134_462957_object114:head# 2024-04-25T09:31:49.315 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.356 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:86e69dbb:::benchmark_data_smithi134_462957_object334:head# 2024-04-25T09:31:49.356 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.398 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:87789c1f:::benchmark_data_smithi134_462957_object299:head# 2024-04-25T09:31:49.398 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.440 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:87e9176e:::benchmark_data_smithi134_462957_object176:head# 2024-04-25T09:31:49.440 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.481 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:88b81233:::benchmark_data_smithi134_462957_object197:head# 2024-04-25T09:31:49.482 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.523 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8a55344d:::benchmark_data_smithi134_462957_object163:head# 2024-04-25T09:31:49.524 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.565 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8b35579f:::benchmark_data_smithi134_462957_object101:head# 2024-04-25T09:31:49.565 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.614 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c42d79e:::benchmark_data_smithi134_462957_object285:head# 2024-04-25T09:31:49.614 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.674 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c5cb74b:::benchmark_data_smithi134_462957_object30:head# 2024-04-25T09:31:49.674 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.724 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c777440:::benchmark_data_smithi134_462957_object69:head# 2024-04-25T09:31:49.724 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.766 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8d18392e:::benchmark_data_smithi134_462957_object42:head# 2024-04-25T09:31:49.766 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.808 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8d4a51df:::benchmark_data_smithi134_462957_object202:head# 2024-04-25T09:31:49.808 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.849 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8f9fc238:::benchmark_data_smithi134_462957_object222:head# 2024-04-25T09:31:49.849 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.891 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8fcd40aa:::benchmark_data_smithi134_462957_object13:head# 2024-04-25T09:31:49.891 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.933 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8ffd53b9:::benchmark_data_smithi134_462957_object152:head# 2024-04-25T09:31:49.933 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:49.975 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:912ceabf:::benchmark_data_smithi134_462957_object291:head# 2024-04-25T09:31:49.975 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.016 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:913e15cc:::benchmark_data_smithi134_462957_object283:head# 2024-04-25T09:31:50.017 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.100 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:92bf16d1:::benchmark_data_smithi134_462957_object195:head# 2024-04-25T09:31:50.100 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.167 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:92ed1f0f:::benchmark_data_smithi134_462957_object65:head# 2024-04-25T09:31:50.167 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.243 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9417108d:::benchmark_data_smithi134_462957_object261:head# 2024-04-25T09:31:50.243 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.284 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:954cc32c:::benchmark_data_smithi134_462957_object139:head# 2024-04-25T09:31:50.284 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.326 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9632b147:::benchmark_data_smithi134_462957_object154:head# 2024-04-25T09:31:50.326 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.368 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9726b3f5:::benchmark_data_smithi134_462957_object135:head# 2024-04-25T09:31:50.368 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.410 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:97aa80c9:::benchmark_data_smithi134_462957_object210:head# 2024-04-25T09:31:50.410 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.451 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9870b61d:::benchmark_data_smithi134_462957_object50:head# 2024-04-25T09:31:50.452 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.493 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9a3c902a:::benchmark_data_smithi134_462957_object66:head# 2024-04-25T09:31:50.493 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.535 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9a63a41b:::benchmark_data_smithi134_462957_object59:head# 2024-04-25T09:31:50.535 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.577 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9af4c21c:::benchmark_data_smithi134_462957_object99:head# 2024-04-25T09:31:50.577 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.619 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9c02c888:::benchmark_data_smithi134_462957_object100:head# 2024-04-25T09:31:50.619 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.660 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9d740ac7:::benchmark_data_smithi134_462957_object156:head# 2024-04-25T09:31:50.660 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.702 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e32be9a:::benchmark_data_smithi134_462957_object54:head# 2024-04-25T09:31:50.702 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.744 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e6056cd:::benchmark_data_smithi134_462957_object164:head# 2024-04-25T09:31:50.744 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.810 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9ebf06db:::benchmark_data_smithi134_462957_object284:head# 2024-04-25T09:31:50.810 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.853 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a045e8d8:::benchmark_data_smithi134_462957_object234:head# 2024-04-25T09:31:50.853 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.894 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a08b0a4d:::benchmark_data_smithi134_462957_object241:head# 2024-04-25T09:31:50.894 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.936 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a0c022f8:::benchmark_data_smithi134_462957_object226:head# 2024-04-25T09:31:50.936 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:50.978 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a1658627:::benchmark_data_smithi134_462957_object327:head# 2024-04-25T09:31:50.978 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.020 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a1bb4fbf:::benchmark_data_smithi134_462957_object198:head# 2024-04-25T09:31:51.020 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.078 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a2b4a5bd:::benchmark_data_smithi134_462957_object27:head# 2024-04-25T09:31:51.078 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.120 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a5ca7c00:::benchmark_data_smithi134_462957_object72:head# 2024-04-25T09:31:51.120 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.162 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a5f62c63:::benchmark_data_smithi134_462957_object316:head# 2024-04-25T09:31:51.162 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.203 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a63f834f:::benchmark_data_smithi134_462957_object157:head# 2024-04-25T09:31:51.203 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.245 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a715009b:::benchmark_data_smithi134_462957_object169:head# 2024-04-25T09:31:51.245 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.287 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a85dbb56:::benchmark_data_smithi134_462957_object307:head# 2024-04-25T09:31:51.287 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.329 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a8d1cd5f:::benchmark_data_smithi134_462957_object107:head# 2024-04-25T09:31:51.329 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.379 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a9621e84:::benchmark_data_smithi134_462957_object303:head# 2024-04-25T09:31:51.379 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.421 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a9a511c7:::benchmark_data_smithi134_462957_object321:head# 2024-04-25T09:31:51.421 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.463 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ab6edda1:::benchmark_data_smithi134_462957_object212:head# 2024-04-25T09:31:51.463 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.504 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:aba338b0:::benchmark_data_smithi134_462957_object184:head# 2024-04-25T09:31:51.504 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.546 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:abdd83aa:::benchmark_data_smithi134_462957_object85:head# 2024-04-25T09:31:51.547 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.588 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ac85576b:::benchmark_data_smithi134_462957_object282:head# 2024-04-25T09:31:51.588 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.664 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ae25ce97:::benchmark_data_smithi134_462957_object75:head# 2024-04-25T09:31:51.664 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.705 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b1f8b3b5:::benchmark_data_smithi134_462957_object160:head# 2024-04-25T09:31:51.705 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.747 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b249419b:::benchmark_data_smithi134_462957_object275:head# 2024-04-25T09:31:51.747 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.780 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b289940b:::benchmark_data_smithi134_462957_object26:head# 2024-04-25T09:31:51.781 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.814 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b29d1223:::benchmark_data_smithi134_462957_object167:head# 2024-04-25T09:31:51.814 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.864 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b4ae5688:::benchmark_data_smithi134_462957_object182:head# 2024-04-25T09:31:51.864 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.897 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b5565581:::benchmark_data_smithi134_462957_object22:head# 2024-04-25T09:31:51.897 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.939 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b56627a7:::benchmark_data_smithi134_462957_object108:head# 2024-04-25T09:31:51.939 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:51.981 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b6ff86cd:::benchmark_data_smithi134_462957_object166:head# 2024-04-25T09:31:51.981 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.023 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b7edf419:::benchmark_data_smithi134_462957_object84:head# 2024-04-25T09:31:52.023 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.090 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:31:52.090 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.148 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b8f154ff:::benchmark_data_smithi134_462957_object206:head# 2024-04-25T09:31:52.148 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.199 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b93b7738:::benchmark_data_smithi134_462957_object92:head# 2024-04-25T09:31:52.199 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.240 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b997874c:::benchmark_data_smithi134_462957_object320:head# 2024-04-25T09:31:52.241 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.282 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ba34c80e:::benchmark_data_smithi134_462957_object29:head# 2024-04-25T09:31:52.282 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.324 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bb51ab09:::benchmark_data_smithi134_462957_object246:head# 2024-04-25T09:31:52.324 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.366 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bb9ed292:::benchmark_data_smithi134_462957_object44:head# 2024-04-25T09:31:52.366 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.408 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bc0fe2eb:::benchmark_data_smithi134_462957_object105:head# 2024-04-25T09:31:52.408 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.449 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bd33c13c:::benchmark_data_smithi134_462957_object121:head# 2024-04-25T09:31:52.449 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.491 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bd52ec16:::benchmark_data_smithi134_462957_object68:head# 2024-04-25T09:31:52.491 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.533 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:be19e778:::benchmark_data_smithi134_462957_object238:head# 2024-04-25T09:31:52.533 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.575 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bf64a100:::benchmark_data_smithi134_462957_object83:head# 2024-04-25T09:31:52.575 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.616 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c03f464f:::benchmark_data_smithi134_462957_object18:head# 2024-04-25T09:31:52.616 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.658 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c0f283ce:::benchmark_data_smithi134_462957_object124:head# 2024-04-25T09:31:52.658 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.700 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c16c7b78:::benchmark_data_smithi134_462957_object2:head# 2024-04-25T09:31:52.700 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.742 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c18f6438:::benchmark_data_smithi134_462957_object220:head# 2024-04-25T09:31:52.742 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.783 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c244846a:::benchmark_data_smithi134_462957_object329:head# 2024-04-25T09:31:52.784 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c274bcc9:::benchmark_data_smithi134_462957_object332:head# 2024-04-25T09:31:52.825 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.867 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c2e9f6b4:::benchmark_data_smithi134_462957_object330:head# 2024-04-25T09:31:52.867 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.909 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c35cfddf:::benchmark_data_smithi134_462957_object200:head# 2024-04-25T09:31:52.909 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.951 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c47b78d0:::benchmark_data_smithi134_462957_object117:head# 2024-04-25T09:31:52.951 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:52.992 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c70a7496:::benchmark_data_smithi134_462957_object60:head# 2024-04-25T09:31:52.993 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.034 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c73308dd:::benchmark_data_smithi134_462957_object258:head# 2024-04-25T09:31:53.034 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.076 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c74b145d:::benchmark_data_smithi134_462957_object146:head# 2024-04-25T09:31:53.076 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.118 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c7d37ef1:::benchmark_data_smithi134_462957_object125:head# 2024-04-25T09:31:53.118 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.159 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c84d6c63:::benchmark_data_smithi134_462957_object221:head# 2024-04-25T09:31:53.160 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.201 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c8d637f7:::benchmark_data_smithi134_462957_object306:head# 2024-04-25T09:31:53.202 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.243 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c9e347b1:::benchmark_data_smithi134_462957_object242:head# 2024-04-25T09:31:53.243 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.285 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ca46b863:::benchmark_data_smithi134_462957_object57:head# 2024-04-25T09:31:53.285 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.327 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ca795a74:::benchmark_data_smithi134_462957_object81:head# 2024-04-25T09:31:53.327 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.369 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cc49d25d:::benchmark_data_smithi134_462957_object144:head# 2024-04-25T09:31:53.369 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.410 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ccd8cbfe:::benchmark_data_smithi134_462957_object5:head# 2024-04-25T09:31:53.410 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.452 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cce05002:::benchmark_data_smithi134_462957_object130:head# 2024-04-25T09:31:53.452 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.494 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cd79c5f8:::benchmark_data_smithi134_462957_object76:head# 2024-04-25T09:31:53.494 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.536 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cdb281d5:::benchmark_data_smithi134_462957_object46:head# 2024-04-25T09:31:53.536 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.577 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cdc69e54:::benchmark_data_smithi134_462957_object252:head# 2024-04-25T09:31:53.578 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.653 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf2a4062:::benchmark_data_smithi134_462957_object199:head# 2024-04-25T09:31:53.653 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.711 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf35812c:::benchmark_data_smithi134_462957_object243:head# 2024-04-25T09:31:53.711 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.753 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf693ea1:::benchmark_data_smithi134_462957_object147:head# 2024-04-25T09:31:53.753 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.795 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d0dad50c:::benchmark_data_smithi134_462957_object97:head# 2024-04-25T09:31:53.795 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.837 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d1761cb1:::benchmark_data_smithi134_462957_object251:head# 2024-04-25T09:31:53.837 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.878 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d1b7fc50:::benchmark_data_smithi134_462957_object211:head# 2024-04-25T09:31:53.879 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.920 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d2bd3105:::benchmark_data_smithi134_462957_object277:head# 2024-04-25T09:31:53.920 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:53.962 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d3201ed2:::benchmark_data_smithi134_462957_object45:head# 2024-04-25T09:31:53.962 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.012 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d423debd:::benchmark_data_smithi134_462957_object115:head# 2024-04-25T09:31:54.012 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.079 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d4c4f287:::benchmark_data_smithi134_462957_object21:head# 2024-04-25T09:31:54.079 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.146 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d50a78a1:::benchmark_data_smithi134_462957_object25:head# 2024-04-25T09:31:54.146 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.222 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d5af3bd1:::benchmark_data_smithi134_462957_object180:head# 2024-04-25T09:31:54.222 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.263 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d5ef9c20:::benchmark_data_smithi134_462957_object70:head# 2024-04-25T09:31:54.264 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.305 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d65dc781:::benchmark_data_smithi134_462957_object323:head# 2024-04-25T09:31:54.306 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.347 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6872c2f:::benchmark_data_smithi134_462957_object192:head# 2024-04-25T09:31:54.347 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.389 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6cbb661:::benchmark_data_smithi134_462957_object279:head# 2024-04-25T09:31:54.389 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.431 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6d8a7de:::benchmark_data_smithi134_462957_object175:head# 2024-04-25T09:31:54.431 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.472 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d9df41a0:::benchmark_data_smithi134_462957_object288:head# 2024-04-25T09:31:54.472 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.514 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dba2aa95:::benchmark_data_smithi134_462957_object31:head# 2024-04-25T09:31:54.514 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.556 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dc540d1b:::benchmark_data_smithi134_462957_object189:head# 2024-04-25T09:31:54.556 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.597 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dc6e92dd:::benchmark_data_smithi134_462957_object159:head# 2024-04-25T09:31:54.598 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.663 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dcca6922:::benchmark_data_smithi134_462957_object311:head# 2024-04-25T09:31:54.663 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.706 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd1204a0:::benchmark_data_smithi134_462957_object71:head# 2024-04-25T09:31:54.706 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.756 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd3e6d82:::benchmark_data_smithi134_462957_object262:head# 2024-04-25T09:31:54.756 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.798 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dfc1c8cf:::benchmark_data_smithi134_462957_object229:head# 2024-04-25T09:31:54.798 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.840 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e06b8840:::benchmark_data_smithi134_462957_object91:head# 2024-04-25T09:31:54.840 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.882 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e06cab0a:::benchmark_data_smithi134_462957_object23:head# 2024-04-25T09:31:54.882 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.923 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e28f6f75:::benchmark_data_smithi134_462957_object300:head# 2024-04-25T09:31:54.924 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.965 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e2be0804:::benchmark_data_smithi134_462957_object235:head# 2024-04-25T09:31:54.965 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:54.998 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e2c8c0bf:::benchmark_data_smithi134_462957_object40:head# 2024-04-25T09:31:54.999 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.032 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e3a40c39:::benchmark_data_smithi134_462957_object63:head# 2024-04-25T09:31:55.032 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.065 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e5228945:::benchmark_data_smithi134_462957_object293:head# 2024-04-25T09:31:55.066 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.099 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e614399c:::benchmark_data_smithi134_462957_object110:head# 2024-04-25T09:31:55.099 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.132 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e65f056d:::benchmark_data_smithi134_462957_object196:head# 2024-04-25T09:31:55.132 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.174 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e6a6c722:::benchmark_data_smithi134_462957_object264:head# 2024-04-25T09:31:55.174 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.208 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e74bd302:::benchmark_data_smithi134_462957_object131:head# 2024-04-25T09:31:55.208 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.249 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e7aa26a1:::benchmark_data_smithi134_462957_object129:head# 2024-04-25T09:31:55.249 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.291 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e8378195:::benchmark_data_smithi134_462957_object256:head# 2024-04-25T09:31:55.291 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.333 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e8ce525c:::benchmark_data_smithi134_462957_object155:head# 2024-04-25T09:31:55.333 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.375 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e8f5abd5:::benchmark_data_smithi134_462957_object56:head# 2024-04-25T09:31:55.375 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.416 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e985cf7e:::benchmark_data_smithi134_462957_object287:head# 2024-04-25T09:31:55.417 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.458 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e9f08f42:::benchmark_data_smithi134_462957_object187:head# 2024-04-25T09:31:55.458 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.500 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ea484b6f:::benchmark_data_smithi134_462957_object82:head# 2024-04-25T09:31:55.500 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.542 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ee73ec6d:::benchmark_data_smithi134_462957_object33:head# 2024-04-25T09:31:55.542 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.584 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f01022da:::benchmark_data_smithi134_462957_object324:head# 2024-04-25T09:31:55.584 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.659 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0800d50:::benchmark_data_smithi134_462957_object217:head# 2024-04-25T09:31:55.659 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.709 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0f00dfc:::benchmark_data_smithi134_462957_object185:head# 2024-04-25T09:31:55.709 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.751 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0f2cea2:::benchmark_data_smithi134_462957_object317:head# 2024-04-25T09:31:55.751 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.793 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f131ea49:::benchmark_data_smithi134_462957_object15:head# 2024-04-25T09:31:55.793 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.835 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f3c72ea5:::benchmark_data_smithi134_462957_object173:head# 2024-04-25T09:31:55.835 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.876 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f581c1b4:::benchmark_data_smithi134_462957_object218:head# 2024-04-25T09:31:55.876 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.918 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f63f8b4d:::benchmark_data_smithi134_462957_object247:head# 2024-04-25T09:31:55.918 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:55.960 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f70159a7:::benchmark_data_smithi134_462957_object208:head# 2024-04-25T09:31:55.960 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.002 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f741ec8b:::benchmark_data_smithi134_462957_object325:head# 2024-04-25T09:31:56.002 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.055 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f7f2b0bd:::benchmark_data_smithi134_462957_object302:head# 2024-04-25T09:31:56.055 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.119 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f8d2618b:::benchmark_data_smithi134_462957_object255:head# 2024-04-25T09:31:56.119 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.177 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f8f97fba:::benchmark_data_smithi134_462957_object178:head# 2024-04-25T09:31:56.178 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.236 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fa5b5d21:::benchmark_data_smithi134_462957_object274:head# 2024-04-25T09:31:56.236 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.278 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:faa372e4:::benchmark_data_smithi134_462957_object228:head# 2024-04-25T09:31:56.278 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.320 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fb20bb8f:::benchmark_data_smithi134_462957_object190:head# 2024-04-25T09:31:56.320 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.362 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fc414211:::benchmark_data_smithi134_462957_object79:head# 2024-04-25T09:31:56.362 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.403 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fc7f33f8:::benchmark_data_smithi134_462957_object37:head# 2024-04-25T09:31:56.403 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.445 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fee7f4ff:::benchmark_data_smithi134_462957_object204:head# 2024-04-25T09:31:56.445 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.487 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ff13f1d8:::benchmark_data_smithi134_462957_object49:head# 2024-04-25T09:31:56.487 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:56.537 INFO:tasks.workunit.client.0.smithi134.stdout:write_pg epoch 9 info 1.1( v 9'170 (0'0,9'170] local-lis/les=7/8 n=170 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:31:56.579 INFO:tasks.workunit.client.0.smithi134.stdout:Import successful 2024-04-25T09:31:57.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:114: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:31:58.483 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e20 pg_nums {1={7=2,19=1}} deleted_pools ) 2024-04-25T09:31:58.483 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.0 2024-04-25T09:31:58.483 INFO:tasks.workunit.client.0.smithi134.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:31:58.483 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:014fbc01:::benchmark_data_smithi134_462957_object116:head# 2024-04-25T09:31:58.483 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.584 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:01ae4a14:::benchmark_data_smithi134_462957_object335:head# 2024-04-25T09:31:58.584 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.626 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:022eaaef:::benchmark_data_smithi134_462957_object132:head# 2024-04-25T09:31:58.626 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.667 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:04df5908:::benchmark_data_smithi134_462957_object10:head# 2024-04-25T09:31:58.667 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.709 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:05e918e6:::benchmark_data_smithi134_462957_object87:head# 2024-04-25T09:31:58.709 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.751 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:08057fd0:::benchmark_data_smithi134_462957_object95:head# 2024-04-25T09:31:58.751 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.793 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:08313bb1:::benchmark_data_smithi134_462957_object240:head# 2024-04-25T09:31:58.793 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.834 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0958de82:::benchmark_data_smithi134_462957_object305:head# 2024-04-25T09:31:58.835 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.876 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:09655936:::benchmark_data_smithi134_462957_object278:head# 2024-04-25T09:31:58.876 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.918 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0a7e421b:::benchmark_data_smithi134_462957_object39:head# 2024-04-25T09:31:58.918 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:58.960 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0be5f62e:::benchmark_data_smithi134_462957_object126:head# 2024-04-25T09:31:58.960 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.002 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0ce6c2ff:::benchmark_data_smithi134_462957_object48:head# 2024-04-25T09:31:59.002 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.043 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0d00a542:::benchmark_data_smithi134_462957_object112:head# 2024-04-25T09:31:59.043 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.085 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0d256506:::benchmark_data_smithi134_462957_object177:head# 2024-04-25T09:31:59.085 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.127 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0e18ad9b:::benchmark_data_smithi134_462957_object52:head# 2024-04-25T09:31:59.127 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.160 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0f50edf7:::benchmark_data_smithi134_462957_object304:head# 2024-04-25T09:31:59.160 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.194 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0f680139:::benchmark_data_smithi134_462957_object310:head# 2024-04-25T09:31:59.194 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.227 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0fc29821:::benchmark_data_smithi134_462957_object276:head# 2024-04-25T09:31:59.227 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.277 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:109d776b:::benchmark_data_smithi134_462957_object331:head# 2024-04-25T09:31:59.277 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.311 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1131e4a2:::benchmark_data_smithi134_462957_object266:head# 2024-04-25T09:31:59.311 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.344 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:11a2460d:::benchmark_data_smithi134_462957_object267:head# 2024-04-25T09:31:59.344 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.378 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1255ae71:::benchmark_data_smithi134_462957_object123:head# 2024-04-25T09:31:59.378 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.411 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:134c0c2c:::benchmark_data_smithi134_462957_object140:head# 2024-04-25T09:31:59.411 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.445 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:136d3c79:::benchmark_data_smithi134_462957_object216:head# 2024-04-25T09:31:59.445 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.487 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:14c28473:::benchmark_data_smithi134_462957_object143:head# 2024-04-25T09:31:59.487 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.520 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:14d46dea:::benchmark_data_smithi134_462957_object322:head# 2024-04-25T09:31:59.520 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.554 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:151932bc:::benchmark_data_smithi134_462957_object127:head# 2024-04-25T09:31:59.554 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.587 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:15788a9d:::benchmark_data_smithi134_462957_object215:head# 2024-04-25T09:31:59.587 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.662 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1607bd5a:::benchmark_data_smithi134_462957_object24:head# 2024-04-25T09:31:59.663 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.712 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:16357f0a:::benchmark_data_smithi134_462957_object16:head# 2024-04-25T09:31:59.712 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.746 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:16baa595:::benchmark_data_smithi134_462957_object309:head# 2024-04-25T09:31:59.746 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.788 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1965afdf:::benchmark_data_smithi134_462957_object89:head# 2024-04-25T09:31:59.788 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.829 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1affb4ca:::benchmark_data_smithi134_462957_object62:head# 2024-04-25T09:31:59.829 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.871 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1b230e12:::benchmark_data_smithi134_462957_object118:head# 2024-04-25T09:31:59.871 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.913 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1bd04fc7:::benchmark_data_smithi134_462957_object188:head# 2024-04-25T09:31:59.913 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.955 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1d14325e:::benchmark_data_smithi134_462957_object41:head# 2024-04-25T09:31:59.955 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:31:59.997 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1d8f10c1:::benchmark_data_smithi134_462957_object86:head# 2024-04-25T09:31:59.997 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.056 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1da2d015:::benchmark_data_smithi134_462957_object64:head# 2024-04-25T09:32:00.056 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.122 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1dad9b9c:::benchmark_data_smithi134_462957_object257:head# 2024-04-25T09:32:00.122 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.189 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1fb02fa0:::benchmark_data_smithi134_462957_object193:head# 2024-04-25T09:32:00.189 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.239 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:203c2195:::benchmark_data_smithi134_462957_object254:head# 2024-04-25T09:32:00.239 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.289 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:22302b8d:::benchmark_data_smithi134_462957_object80:head# 2024-04-25T09:32:00.290 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.331 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:228f6133:::benchmark_data_smithi134_462957_object74:head# 2024-04-25T09:32:00.331 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.373 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:23e9d7b5:::benchmark_data_smithi134_462957_object245:head# 2024-04-25T09:32:00.374 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.415 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2426c803:::benchmark_data_smithi134_462957_object290:head# 2024-04-25T09:32:00.415 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.457 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:244ef85c:::benchmark_data_smithi134_462957_object170:head# 2024-04-25T09:32:00.457 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.498 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2513bbe2:::benchmark_data_smithi134_462957_object207:head# 2024-04-25T09:32:00.499 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.542 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2568f8b5:::benchmark_data_smithi134_462957_object8:head# 2024-04-25T09:32:00.542 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.582 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:260f1533:::benchmark_data_smithi134_462957_object191:head# 2024-04-25T09:32:00.582 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.624 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:27170e69:::benchmark_data_smithi134_462957_object34:head# 2024-04-25T09:32:00.624 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.666 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:275fd7c3:::benchmark_data_smithi134_462957_object225:head# 2024-04-25T09:32:00.666 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.708 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:28ef7a99:::benchmark_data_smithi134_462957_object162:head# 2024-04-25T09:32:00.708 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.749 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2b3177c6:::benchmark_data_smithi134_462957_object298:head# 2024-04-25T09:32:00.749 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.791 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2b9356c2:::benchmark_data_smithi134_462957_object270:head# 2024-04-25T09:32:00.791 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.833 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2cb634ef:::benchmark_data_smithi134_462957_object297:head# 2024-04-25T09:32:00.833 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.875 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2ccb43ac:::benchmark_data_smithi134_462957_object28:head# 2024-04-25T09:32:00.875 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.916 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2cf40efb:::benchmark_data_smithi134_462957_object289:head# 2024-04-25T09:32:00.917 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:00.958 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2d9f151f:::benchmark_data_smithi134_462957_object73:head# 2024-04-25T09:32:00.959 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.000 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2dfb304e:::benchmark_data_smithi134_462957_object138:head# 2024-04-25T09:32:01.000 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.042 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2e76756e:::benchmark_data_smithi134_462957_object174:head# 2024-04-25T09:32:01.042 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.083 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2f2a62da:::benchmark_data_smithi134_462957_object230:head# 2024-04-25T09:32:01.084 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.125 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2f954ef6:::benchmark_data_smithi134_462957_object120:head# 2024-04-25T09:32:01.125 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.167 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:302de82f:::benchmark_data_smithi134_462957_object268:head# 2024-04-25T09:32:01.167 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.209 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:314aacf8:::benchmark_data_smithi134_462957_object98:head# 2024-04-25T09:32:01.209 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.251 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:328e347c:::benchmark_data_smithi134_462957_object203:head# 2024-04-25T09:32:01.251 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.309 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:33056cad:::benchmark_data_smithi134_462957_object248:head# 2024-04-25T09:32:01.309 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.351 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3342be62:::benchmark_data_smithi134_462957_object158:head# 2024-04-25T09:32:01.351 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.393 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:33c19275:::benchmark_data_smithi134_462957_object43:head# 2024-04-25T09:32:01.393 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.434 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:34187219:::benchmark_data_smithi134_462957_object294:head# 2024-04-25T09:32:01.434 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.476 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:344b976e:::benchmark_data_smithi134_462957_object231:head# 2024-04-25T09:32:01.476 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.518 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:358811ac:::benchmark_data_smithi134_462957_object142:head# 2024-04-25T09:32:01.518 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.563 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:361e89fa:::benchmark_data_smithi134_462957_object308:head# 2024-04-25T09:32:01.563 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.623 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:369c019f:::benchmark_data_smithi134_462957_object313:head# 2024-04-25T09:32:01.623 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.685 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:39249c23:::benchmark_data_smithi134_462957_object161:head# 2024-04-25T09:32:01.685 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.744 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3a8ee0cf:::benchmark_data_smithi134_462957_object151:head# 2024-04-25T09:32:01.744 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.786 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3b9ee75a:::benchmark_data_smithi134_462957_object326:head# 2024-04-25T09:32:01.786 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.827 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3e5f4585:::benchmark_data_smithi134_462957_object47:head# 2024-04-25T09:32:01.827 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.869 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3ed5c4c3:::benchmark_data_smithi134_462957_object205:head# 2024-04-25T09:32:01.869 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.911 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3fc6ca95:::benchmark_data_smithi134_462957_object12:head# 2024-04-25T09:32:01.911 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.953 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:40470fd1:::benchmark_data_smithi134_462957_object17:head# 2024-04-25T09:32:01.953 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:01.995 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:417961ce:::benchmark_data_smithi134_462957_object319:head# 2024-04-25T09:32:01.995 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.048 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:418ec0ee:::benchmark_data_smithi134_462957_object149:head# 2024-04-25T09:32:02.048 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.120 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:422d92f6:::benchmark_data_smithi134_462957_object272:head# 2024-04-25T09:32:02.120 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.187 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:435e2086:::benchmark_data_smithi134_462957_object171:head# 2024-04-25T09:32:02.187 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.237 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:444a49fd:::benchmark_data_smithi134_462957_object328:head# 2024-04-25T09:32:02.237 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.271 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:44a8dd41:::benchmark_data_smithi134_462957_object3:head# 2024-04-25T09:32:02.271 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.304 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:467046b5:::benchmark_data_smithi134_462957_object265:head# 2024-04-25T09:32:02.304 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.371 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4672b82d:::benchmark_data_smithi134_462957_object295:head# 2024-04-25T09:32:02.371 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.405 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47196e71:::benchmark_data_smithi134_462957_object269:head# 2024-04-25T09:32:02.405 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.438 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:478e238f:::benchmark_data_smithi134_462957_object213:head# 2024-04-25T09:32:02.438 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.471 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47e25b7f:::benchmark_data_smithi134_462957_object249:head# 2024-04-25T09:32:02.471 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.505 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47f5dd1f:::benchmark_data_smithi134_462957_object122:head# 2024-04-25T09:32:02.505 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.538 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:49e180d1:::benchmark_data_smithi134_462957_object88:head# 2024-04-25T09:32:02.538 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.572 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4a676de3:::benchmark_data_smithi134_462957_object263:head# 2024-04-25T09:32:02.572 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.605 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4b79b182:::benchmark_data_smithi134_462957_object137:head# 2024-04-25T09:32:02.605 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.638 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4cb6f1b7:::benchmark_data_smithi134_462957_object78:head# 2024-04-25T09:32:02.639 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.672 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4e62c3c1:::benchmark_data_smithi134_462957_object53:head# 2024-04-25T09:32:02.672 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.714 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4e981d63:::benchmark_data_smithi134_462957_object227:head# 2024-04-25T09:32:02.714 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.756 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4fda476d:::benchmark_data_smithi134_462957_object194:head# 2024-04-25T09:32:02.756 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.806 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:504c6844:::benchmark_data_smithi134_462957_object94:head# 2024-04-25T09:32:02.806 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.848 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:504ee13c:::benchmark_data_smithi134_462957_object93:head# 2024-04-25T09:32:02.848 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.889 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:511d8705:::benchmark_data_smithi134_462957_object271:head# 2024-04-25T09:32:02.890 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.931 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:51b089df:::benchmark_data_smithi134_462957_object273:head# 2024-04-25T09:32:02.932 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:02.973 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5266c200:::benchmark_data_smithi134_462957_object32:head# 2024-04-25T09:32:02.973 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.015 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:542d36c1:::benchmark_data_smithi134_462957_object4:head# 2024-04-25T09:32:03.015 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.056 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:544671af:::benchmark_data_smithi134_462957_object172:head# 2024-04-25T09:32:03.056 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.098 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:549eba21:::benchmark_data_smithi134_462957_object133:head# 2024-04-25T09:32:03.098 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.140 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:556d8a79:::benchmark_data_smithi134_462957_object214:head# 2024-04-25T09:32:03.140 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.182 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:556e8b3c:::benchmark_data_smithi134_462957_object96:head# 2024-04-25T09:32:03.182 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.223 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:55950821:::benchmark_data_smithi134_462957_object141:head# 2024-04-25T09:32:03.224 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.265 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:575ccb6b:::benchmark_data_smithi134_462957_object280:head# 2024-04-25T09:32:03.266 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.307 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:57a020db:::benchmark_data_smithi134_462957_object286:head# 2024-04-25T09:32:03.307 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.349 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:58587487:::benchmark_data_smithi134_462957_object51:head# 2024-04-25T09:32:03.349 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.391 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5c373486:::benchmark_data_smithi134_462957_object14:head# 2024-04-25T09:32:03.391 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.432 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5dada283:::benchmark_data_smithi134_462957_object77:head# 2024-04-25T09:32:03.433 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.474 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5e48ae44:::benchmark_data_smithi134_462957_object333:head# 2024-04-25T09:32:03.474 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.516 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:60d506d0:::benchmark_data_smithi134_462957_object111:head# 2024-04-25T09:32:03.516 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.558 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:61342236:::benchmark_data_smithi134_462957_object301:head# 2024-04-25T09:32:03.558 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.614 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:61d1b2a4:::benchmark_data_smithi134_462957_object179:head# 2024-04-25T09:32:03.614 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.683 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:621d43c5:::benchmark_data_smithi134_462957_object102:head# 2024-04-25T09:32:03.684 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.725 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:62545d41:::benchmark_data_smithi134_462957_object318:head# 2024-04-25T09:32:03.725 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.767 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:632b2ff2:::benchmark_data_smithi134_462957_object113:head# 2024-04-25T09:32:03.767 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.808 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:63effe0b:::benchmark_data_smithi134_462957_object1:head# 2024-04-25T09:32:03.808 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.850 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:63f38e63:::benchmark_data_smithi134_462957_object314:head# 2024-04-25T09:32:03.850 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.892 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:645caf70:::benchmark_data_smithi134_462957_object9:head# 2024-04-25T09:32:03.892 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.934 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:64cb656a:::benchmark_data_smithi134_462957_object38:head# 2024-04-25T09:32:03.934 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:03.975 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:651d1c11:::benchmark_data_smithi134_462957_object219:head# 2024-04-25T09:32:03.976 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.017 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:65698cf1:::benchmark_data_smithi134_462957_object315:head# 2024-04-25T09:32:04.017 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.101 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:660b1875:::benchmark_data_smithi134_462957_object153:head# 2024-04-25T09:32:04.101 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.210 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:66e4bb20:::benchmark_data_smithi134_462957_object259:head# 2024-04-25T09:32:04.210 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.268 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67406b53:::benchmark_data_smithi134_462957_object168:head# 2024-04-25T09:32:04.269 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.310 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6753f201:::benchmark_data_smithi134_462957_object19:head# 2024-04-25T09:32:04.311 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.352 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67de4446:::benchmark_data_smithi134_462957_object11:head# 2024-04-25T09:32:04.352 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.394 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68431e33:::benchmark_data_smithi134_462957_object6:head# 2024-04-25T09:32:04.394 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.436 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68da1af8:::benchmark_data_smithi134_462957_object224:head# 2024-04-25T09:32:04.436 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.477 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68e29e1b:::benchmark_data_smithi134_462957_object223:head# 2024-04-25T09:32:04.477 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.519 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68ff3868:::benchmark_data_smithi134_462957_object104:head# 2024-04-25T09:32:04.519 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.561 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:69444b58:::benchmark_data_smithi134_462957_object236:head# 2024-04-25T09:32:04.561 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.603 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6bef445e:::benchmark_data_smithi134_462957_object145:head# 2024-04-25T09:32:04.603 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.645 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6c6b0a68:::benchmark_data_smithi134_462957_object106:head# 2024-04-25T09:32:04.645 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.686 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6cb36ab9:::benchmark_data_smithi134_462957_object128:head# 2024-04-25T09:32:04.687 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.728 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6cb830fe:::benchmark_data_smithi134_462957_object281:head# 2024-04-25T09:32:04.728 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.770 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6d038ea7:::benchmark_data_smithi134_462957_object250:head# 2024-04-25T09:32:04.770 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.812 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6dfad7ef:::benchmark_data_smithi134_462957_object253:head# 2024-04-25T09:32:04.812 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.854 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6e8445ed:::benchmark_data_smithi134_462957_object119:head# 2024-04-25T09:32:04.854 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.895 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6ea927e3:::benchmark_data_smithi134_462957_object67:head# 2024-04-25T09:32:04.895 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.937 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6edbe582:::benchmark_data_smithi134_462957_object260:head# 2024-04-25T09:32:04.937 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:04.979 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6f922ade:::benchmark_data_smithi134_462957_object136:head# 2024-04-25T09:32:04.979 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.021 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6fe808b0:::benchmark_data_smithi134_462957_object7:head# 2024-04-25T09:32:05.021 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.063 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:71c24ec6:::benchmark_data_smithi134_462957_object35:head# 2024-04-25T09:32:05.063 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.105 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7280df13:::benchmark_data_smithi134_462957_object109:head# 2024-04-25T09:32:05.105 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.138 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:72b74ac7:::benchmark_data_smithi134_462957_object292:head# 2024-04-25T09:32:05.138 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.171 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:72d020e0:::benchmark_data_smithi134_462957_object20:head# 2024-04-25T09:32:05.171 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.205 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:73570709:::benchmark_data_smithi134_462957_object244:head# 2024-04-25T09:32:05.205 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.238 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:73fa9d47:::benchmark_data_smithi134_462957_object0:head# 2024-04-25T09:32:05.238 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.272 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:76387380:::benchmark_data_smithi134_462957_object103:head# 2024-04-25T09:32:05.272 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.305 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:77a61cee:::benchmark_data_smithi134_462957_object237:head# 2024-04-25T09:32:05.305 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.338 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7826dd0b:::benchmark_data_smithi134_462957_object233:head# 2024-04-25T09:32:05.338 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.372 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7958a491:::benchmark_data_smithi134_462957_object232:head# 2024-04-25T09:32:05.372 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.506 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7a5eed6c:::benchmark_data_smithi134_462957_object55:head# 2024-04-25T09:32:05.506 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.539 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7bed12e2:::benchmark_data_smithi134_462957_object90:head# 2024-04-25T09:32:05.539 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.572 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7c53f004:::benchmark_data_smithi134_462957_object183:head# 2024-04-25T09:32:05.573 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.614 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7cc751e9:::benchmark_data_smithi134_462957_object150:head# 2024-04-25T09:32:05.614 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.673 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7de067cb:::benchmark_data_smithi134_462957_object148:head# 2024-04-25T09:32:05.674 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.706 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7e10a619:::benchmark_data_smithi134_462957_object296:head# 2024-04-25T09:32:05.707 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.740 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7f2f5709:::benchmark_data_smithi134_462957_object58:head# 2024-04-25T09:32:05.740 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.773 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7f4d9c89:::benchmark_data_smithi134_462957_object209:head# 2024-04-25T09:32:05.773 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:32:05.807 INFO:tasks.workunit.client.0.smithi134.stdout:write_pg epoch 9 info 1.0( v 9'167 (0'0,9'167] local-lis/les=7/8 n=167 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:32:05.848 INFO:tasks.workunit.client.0.smithi134.stdout:Import successful 2024-04-25T09:32:06.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:115: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:32:06.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:32:06.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:32:06.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:32:06.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:32:06.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:32:06.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:32:06.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:32:06.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:32:06.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:32:06.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:32:06.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:32:06.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:32:06.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:32:06.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:32:06.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:32:06.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:32:06.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:32:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:32:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:32:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:32:06.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:32:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:32:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:32:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:32:06.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:32:06.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:32:06.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:32:06.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:32:06.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:32:06.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:32:06.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:32:06.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:32:06.393 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:32:06.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:32:06.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:32:06.395 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:32:06.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:32:06.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:32:06.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:32:06.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:32:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:32:06.443+0000 7fbf90f487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:06.432 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:32:06.447+0000 7fbf90f487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:06.434 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:32:06.447+0000 7fbf90f487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:32:06.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:32:06.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:32:06.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:32:06.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:32:06.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:32:06.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:06.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:32:06.826 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:32:06.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:06.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:06.997 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:32:07.011+0000 7fbf90f487c0 -1 Falling back to public interface 2024-04-25T09:32:07.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:08.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:08.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:08.258 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:32:08.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:32:08.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:08.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:08.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:09.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:09.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:09.685 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:32:09.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:32:09.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:09.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:09.990 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:32:10.003+0000 7fbf90f487c0 -1 osd.0 20 log_to_monitors true 2024-04-25T09:32:10.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:32:11.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:32:11.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:32:11.112 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:32:11.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:32:11.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:32:11.112 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:32:11.592 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 23 up_thru 23 down_at 21 last_clean_interval [12,20) [v2:127.0.0.1:6802/1381886146,v1:127.0.0.1:6803/1381886146] [v2:127.0.0.1:6804/1381886146,v1:127.0.0.1:6805/1381886146] exists,up c787144a-5beb-4391-bbea-edfe5c80e442 2024-04-25T09:32:11.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:32:11.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:32:11.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:117: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:32:11.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:32:11.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:32:11.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:32:11.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:32:11.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:32:11.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:32:11.691 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:32:12.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:32:12.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:32:12.092 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:32:12.092 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:32:12.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=98784247811 2024-04-25T09:32:12.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 98784247811 2024-04-25T09:32:12.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-98784247811' 2024-04-25T09:32:12.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:32:12.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-98784247811 2024-04-25T09:32:12.257 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:32:12.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:32:12.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-98784247811 2024-04-25T09:32:12.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:32:12.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=98784247811 2024-04-25T09:32:12.259 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 98784247811 2024-04-25T09:32:12.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 98784247811' 2024-04-25T09:32:12.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:32:12.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 98784247811 2024-04-25T09:32:12.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:32:13.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:32:13.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:32:14.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 98784247811 -lt 98784247811 2024-04-25T09:32:14.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:32:14.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:32:14.047 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:32:14.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:32:14.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:32:14.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:32:14.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:32:14.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:32:14.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:32:14.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:32:14.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:32:15.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:32:15.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:32:15.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:32:15.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:32:15.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:32:15.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:32:15.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:32:15.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: seq 1 50 2024-04-25T09:32:15.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:15.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:16.189 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:16.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:16.950 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:16.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:16.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:17.921 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:17.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:18.573 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:18.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:18.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:19.326 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:19.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:20.204 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:20.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:20.949 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:20.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:21.810 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:21.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:21.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:22.538 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:22.559 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:23.317 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:23.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:23.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:24.195 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:24.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:25.015 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:25.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:25.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:25.843 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:25.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:26.545 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:26.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:26.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:27.374 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:27.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:28.227 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:28.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:28.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:29.022 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:29.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:29.825 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:29.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:29.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:30.519 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:30.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:31.330 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:31.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:31.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:32.167 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:32.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:32.995 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:33.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:33.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:33.799 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:33.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:34.593 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:34.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:34.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:35.414 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:35.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:36.450 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:36.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:36.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:37.422 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:37.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:38.218 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:38.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:38.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:38.936 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:38.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:39.731 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:39.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:39.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:40.442 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:40.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:41.295 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:41.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:41.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:42.057 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:42.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:42.877 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:42.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:42.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:43.580 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:43.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:44.257 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:44.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:44.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:45.127 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:45.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:45.922 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:45.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:45.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:46.742 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:46.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:47.696 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:47.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:47.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:48.432 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:48.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:49.302 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:49.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:49.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:50.189 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:50.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:51.009 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:51.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:51.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:51.795 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:51.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:52.582 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:52.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:52.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:53.409 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:53.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:54.088 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:54.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:54.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:54.932 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:54.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:55.828 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:55.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:55.850 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:56.455 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:56.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:57.174 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:57.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:57.197 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:58.128 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:58.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:32:58.940 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:32:58.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:32:58.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:32:59.751 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:32:59.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:00.403 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:00.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:00.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:01.181 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:01.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:01.943 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:01.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:01.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:02.779 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:02.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:03.574 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:03.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:03.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:04.443 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:04.464 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:05.272 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:05.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:05.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:05.991 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:06.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:06.794 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:06.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:06.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:07.618 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:07.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:08.383 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:08.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:08.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:09.195 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:09.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:09.981 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:10.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:10.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:10.767 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:10.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:11.612 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:11.634 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:11.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:12.398 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:12.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:13.125 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:13.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:13.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:13.937 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:13.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:14.731 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:14.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:14.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:15.567 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:15.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:16.388 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:16.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:16.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:17.157 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:17.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:18.136 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:18.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:18.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:18.889 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:18.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:19.793 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:19.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:19.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:20.529 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:20.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:21.265 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:21.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:21.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:22.077 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:22.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:22.880 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:22.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:22.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:23.649 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:23.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:24.377 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:24.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:24.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:25.213 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:25.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:25.983 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:26.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:26.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:26.853 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:26.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:27.623 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:27.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:27.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:28.334 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:28.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:29.129 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:29.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:29.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:29.923 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:29.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:30.794 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:30.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:30.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:31.563 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:31.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:32.316 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:32.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:32.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:33.220 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:33.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:33.905 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:33.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:120: TEST_import_after_merge_and_gap: for f in `seq 1 50` 2024-04-25T09:33:33.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:121: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:34.759 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:34.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:122: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:35.520 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:35.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:127: TEST_import_after_merge_and_gap: rados -p foo bench 1 write -b 1024 --no-cleanup 2024-04-25T09:33:35.592 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T09:33:35.592 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 1 seconds or 0 objects 2024-04-25T09:33:35.592 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_468469 2024-04-25T09:33:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:33:35.592 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:33:36.592 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 145 129 0.125958 0.125977 0.0752981 0.116107 2024-04-25T09:33:37.592 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 1.04103 2024-04-25T09:33:37.592 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 145 2024-04-25T09:33:37.592 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 1024 2024-04-25T09:33:37.592 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 1024 2024-04-25T09:33:37.592 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 0.136021 2024-04-25T09:33:37.592 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 0 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 0.125977 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 0.125977 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 139 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 0 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 129 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 129 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 0.11153 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.0416228 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 0.180183 2024-04-25T09:33:37.593 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.0666776 2024-04-25T09:33:37.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:128: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:33:37.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:33:37.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:33:37.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:33:37.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:33:37.680 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:33:37.681 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:33:37.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:33:37.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:33:37.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:33:37.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:33:37.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:33:37.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:33:37.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:33:37.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:33:37.801 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:33:38.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:33:38.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:33:38.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:33:38.420 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:33:38.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=98784247830 2024-04-25T09:33:38.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 98784247830 2024-04-25T09:33:38.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-98784247830' 2024-04-25T09:33:38.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:33:38.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:33:38.591 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-98784247830 2024-04-25T09:33:38.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:33:38.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:33:38.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-98784247830 2024-04-25T09:33:38.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=98784247830 2024-04-25T09:33:38.594 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 98784247830 2024-04-25T09:33:38.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 98784247830' 2024-04-25T09:33:38.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:33:39.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 98784247828 -lt 98784247830 2024-04-25T09:33:39.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:33:40.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:33:40.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:33:40.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 98784247830 -lt 98784247830 2024-04-25T09:33:40.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:33:40.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:33:40.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:33:40.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:33:40.970 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:33:40.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:33:40.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:33:40.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:33:40.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:33:40.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:33:40.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:33:41.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:33:41.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:33:41.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:33:41.398 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:33:41.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:33:41.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:33:41.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:33:41.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:129: TEST_import_after_merge_and_gap: ceph tell osd.0 send_beacon 2024-04-25T09:33:42.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:130: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:33:47.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:131: TEST_import_after_merge_and_gap: ceph osd set nodown 2024-04-25T09:33:47.799 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is set 2024-04-25T09:33:47.821 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:132: TEST_import_after_merge_and_gap: ceph osd unset nodown 2024-04-25T09:33:48.687 INFO:tasks.workunit.client.0.smithi134.stderr:nodown is unset 2024-04-25T09:33:48.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:133: TEST_import_after_merge_and_gap: sleep 5 2024-04-25T09:33:53.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:135: TEST_import_after_merge_and_gap: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:33:53.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:33:53.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:33:53.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:33:53.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:33:53.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:33:54.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:33:54.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:138: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:33:55.170 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-04-25T09:33:55.171 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.1 2024-04-25T09:33:55.171 INFO:tasks.workunit.client.0.smithi134.stderr:PG export's map 9 is older than OSD's oldest_map 61 2024-04-25T09:33:55.171 INFO:tasks.workunit.client.0.smithi134.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:33:55.706 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:140: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export-remove --pgid 1.0 --force --file td/pg-split-merge/1.0.later 2024-04-25T09:33:56.543 INFO:tasks.workunit.client.0.smithi134.stderr:Exporting 1.0 info 1.0( v 224'316 (9'170,224'316] local-lis/les=23/24 n=482 ec=7/7 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-04-25T09:33:56.556 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:00b7a8e8:::benchmark_data_smithi134_468469_object49:head# 2024-04-25T09:33:56.557 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:012cc06f:::benchmark_data_smithi134_468469_object113:head# 2024-04-25T09:33:56.558 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:014fbc01:::benchmark_data_smithi134_462957_object116:head# 2024-04-25T09:33:56.562 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:01ae4a14:::benchmark_data_smithi134_462957_object335:head# 2024-04-25T09:33:56.563 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:022eaaef:::benchmark_data_smithi134_462957_object132:head# 2024-04-25T09:33:56.563 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:031c8e30:::benchmark_data_smithi134_468469_object20:head# 2024-04-25T09:33:56.564 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:038827ab:::benchmark_data_smithi134_468469_object23:head# 2024-04-25T09:33:56.564 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:04df5908:::benchmark_data_smithi134_462957_object10:head# 2024-04-25T09:33:56.565 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:05e918e6:::benchmark_data_smithi134_462957_object87:head# 2024-04-25T09:33:56.566 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:08057fd0:::benchmark_data_smithi134_462957_object95:head# 2024-04-25T09:33:56.566 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:08313bb1:::benchmark_data_smithi134_462957_object240:head# 2024-04-25T09:33:56.567 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:093a7b43:::benchmark_data_smithi134_468469_object40:head# 2024-04-25T09:33:56.567 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0958de82:::benchmark_data_smithi134_462957_object305:head# 2024-04-25T09:33:56.568 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:09655936:::benchmark_data_smithi134_462957_object278:head# 2024-04-25T09:33:56.569 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0a7e421b:::benchmark_data_smithi134_462957_object39:head# 2024-04-25T09:33:56.570 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0b3dcfda:::benchmark_data_smithi134_468469_object83:head# 2024-04-25T09:33:56.570 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0be5f62e:::benchmark_data_smithi134_462957_object126:head# 2024-04-25T09:33:56.571 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0ce6c2ff:::benchmark_data_smithi134_462957_object48:head# 2024-04-25T09:33:56.572 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0d00a542:::benchmark_data_smithi134_462957_object112:head# 2024-04-25T09:33:56.572 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0d256506:::benchmark_data_smithi134_462957_object177:head# 2024-04-25T09:33:56.573 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0e18ad9b:::benchmark_data_smithi134_462957_object52:head# 2024-04-25T09:33:56.574 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0f50edf7:::benchmark_data_smithi134_462957_object304:head# 2024-04-25T09:33:56.574 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0f680139:::benchmark_data_smithi134_462957_object310:head# 2024-04-25T09:33:56.575 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0fc29821:::benchmark_data_smithi134_462957_object276:head# 2024-04-25T09:33:56.576 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:109d776b:::benchmark_data_smithi134_462957_object331:head# 2024-04-25T09:33:56.576 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1131e4a2:::benchmark_data_smithi134_462957_object266:head# 2024-04-25T09:33:56.576 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:11a2460d:::benchmark_data_smithi134_462957_object267:head# 2024-04-25T09:33:56.577 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1214a567:::benchmark_data_smithi134_468469_object13:head# 2024-04-25T09:33:56.578 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1255ae71:::benchmark_data_smithi134_462957_object123:head# 2024-04-25T09:33:56.579 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:131e122e:::benchmark_data_smithi134_468469_object87:head# 2024-04-25T09:33:56.580 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:134c0c2c:::benchmark_data_smithi134_462957_object140:head# 2024-04-25T09:33:56.580 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:136d3c79:::benchmark_data_smithi134_462957_object216:head# 2024-04-25T09:33:56.581 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:14c28473:::benchmark_data_smithi134_462957_object143:head# 2024-04-25T09:33:56.582 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:14d46dea:::benchmark_data_smithi134_462957_object322:head# 2024-04-25T09:33:56.582 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:151932bc:::benchmark_data_smithi134_462957_object127:head# 2024-04-25T09:33:56.583 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:15788a9d:::benchmark_data_smithi134_462957_object215:head# 2024-04-25T09:33:56.584 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1607bd5a:::benchmark_data_smithi134_462957_object24:head# 2024-04-25T09:33:56.585 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:16357f0a:::benchmark_data_smithi134_462957_object16:head# 2024-04-25T09:33:56.585 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:16baa595:::benchmark_data_smithi134_462957_object309:head# 2024-04-25T09:33:56.585 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:177ef5cc:::benchmark_data_smithi134_468469_object41:head# 2024-04-25T09:33:56.587 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:18e81c1e:::benchmark_data_smithi134_468469_object52:head# 2024-04-25T09:33:56.587 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1965afdf:::benchmark_data_smithi134_462957_object89:head# 2024-04-25T09:33:56.588 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1affb4ca:::benchmark_data_smithi134_462957_object62:head# 2024-04-25T09:33:56.590 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1b230e12:::benchmark_data_smithi134_462957_object118:head# 2024-04-25T09:33:56.591 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1bd04fc7:::benchmark_data_smithi134_462957_object188:head# 2024-04-25T09:33:56.592 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1c79513d:::benchmark_data_smithi134_468469_object117:head# 2024-04-25T09:33:56.593 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1d14325e:::benchmark_data_smithi134_462957_object41:head# 2024-04-25T09:33:56.593 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1d1e4cdb:::benchmark_data_smithi134_468469_object45:head# 2024-04-25T09:33:56.594 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1d8f10c1:::benchmark_data_smithi134_462957_object86:head# 2024-04-25T09:33:56.594 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1da2d015:::benchmark_data_smithi134_462957_object64:head# 2024-04-25T09:33:56.595 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1dad9b9c:::benchmark_data_smithi134_462957_object257:head# 2024-04-25T09:33:56.596 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1efb0d17:::benchmark_data_smithi134_468469_object17:head# 2024-04-25T09:33:56.597 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1fb02fa0:::benchmark_data_smithi134_462957_object193:head# 2024-04-25T09:33:56.598 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:203c2195:::benchmark_data_smithi134_462957_object254:head# 2024-04-25T09:33:56.598 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:22302b8d:::benchmark_data_smithi134_462957_object80:head# 2024-04-25T09:33:56.599 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:226a74e1:::benchmark_data_smithi134_468469_object138:head# 2024-04-25T09:33:56.600 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:228f6133:::benchmark_data_smithi134_462957_object74:head# 2024-04-25T09:33:56.600 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:234eb4d4:::benchmark_data_smithi134_468469_object69:head# 2024-04-25T09:33:56.601 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:23e9d7b5:::benchmark_data_smithi134_462957_object245:head# 2024-04-25T09:33:56.602 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:23fa9a3a:::benchmark_data_smithi134_468469_object50:head# 2024-04-25T09:33:56.602 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2426c803:::benchmark_data_smithi134_462957_object290:head# 2024-04-25T09:33:56.603 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:244ef85c:::benchmark_data_smithi134_462957_object170:head# 2024-04-25T09:33:56.604 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2513bbe2:::benchmark_data_smithi134_462957_object207:head# 2024-04-25T09:33:56.605 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2568f8b5:::benchmark_data_smithi134_462957_object8:head# 2024-04-25T09:33:56.605 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:260f1533:::benchmark_data_smithi134_462957_object191:head# 2024-04-25T09:33:56.606 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2671aca6:::benchmark_data_smithi134_468469_object15:head# 2024-04-25T09:33:56.607 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:27170e69:::benchmark_data_smithi134_462957_object34:head# 2024-04-25T09:33:56.607 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:275fd7c3:::benchmark_data_smithi134_462957_object225:head# 2024-04-25T09:33:56.608 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:28ef7a99:::benchmark_data_smithi134_462957_object162:head# 2024-04-25T09:33:56.609 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2a5b63ae:::benchmark_data_smithi134_468469_object77:head# 2024-04-25T09:33:56.609 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2b3177c6:::benchmark_data_smithi134_462957_object298:head# 2024-04-25T09:33:56.610 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2b9356c2:::benchmark_data_smithi134_462957_object270:head# 2024-04-25T09:33:56.611 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2cb634ef:::benchmark_data_smithi134_462957_object297:head# 2024-04-25T09:33:56.611 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2ccb43ac:::benchmark_data_smithi134_462957_object28:head# 2024-04-25T09:33:56.612 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2cf40efb:::benchmark_data_smithi134_462957_object289:head# 2024-04-25T09:33:56.613 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2d9f151f:::benchmark_data_smithi134_462957_object73:head# 2024-04-25T09:33:56.613 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2dfb304e:::benchmark_data_smithi134_462957_object138:head# 2024-04-25T09:33:56.613 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2e76756e:::benchmark_data_smithi134_462957_object174:head# 2024-04-25T09:33:56.614 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2e90f24f:::benchmark_data_smithi134_468469_object124:head# 2024-04-25T09:33:56.615 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2ed12146:::benchmark_data_smithi134_468469_object44:head# 2024-04-25T09:33:56.615 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2f2a62da:::benchmark_data_smithi134_462957_object230:head# 2024-04-25T09:33:56.616 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2f790bda:::benchmark_data_smithi134_468469_object8:head# 2024-04-25T09:33:56.617 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2f954ef6:::benchmark_data_smithi134_462957_object120:head# 2024-04-25T09:33:56.617 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:302de82f:::benchmark_data_smithi134_462957_object268:head# 2024-04-25T09:33:56.618 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:30d87a29:::benchmark_data_smithi134_468469_object29:head# 2024-04-25T09:33:56.619 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:314aacf8:::benchmark_data_smithi134_462957_object98:head# 2024-04-25T09:33:56.619 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:31df473b:::benchmark_data_smithi134_468469_object43:head# 2024-04-25T09:33:56.620 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:328e347c:::benchmark_data_smithi134_462957_object203:head# 2024-04-25T09:33:56.620 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:33056cad:::benchmark_data_smithi134_462957_object248:head# 2024-04-25T09:33:56.621 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3342be62:::benchmark_data_smithi134_462957_object158:head# 2024-04-25T09:33:56.622 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:33c19275:::benchmark_data_smithi134_462957_object43:head# 2024-04-25T09:33:56.622 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:34187219:::benchmark_data_smithi134_462957_object294:head# 2024-04-25T09:33:56.623 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:344b976e:::benchmark_data_smithi134_462957_object231:head# 2024-04-25T09:33:56.624 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:35331aa1:::benchmark_data_smithi134_468469_object72:head# 2024-04-25T09:33:56.625 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:358811ac:::benchmark_data_smithi134_462957_object142:head# 2024-04-25T09:33:56.625 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:35c66d5e:::benchmark_data_smithi134_468469_object75:head# 2024-04-25T09:33:56.626 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:361e89fa:::benchmark_data_smithi134_462957_object308:head# 2024-04-25T09:33:56.626 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:361f3750:::benchmark_data_smithi134_468469_object109:head# 2024-04-25T09:33:56.627 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:369c019f:::benchmark_data_smithi134_462957_object313:head# 2024-04-25T09:33:56.628 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:36a6d606:::benchmark_data_smithi134_468469_object73:head# 2024-04-25T09:33:56.628 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:382b9d8b:::benchmark_data_smithi134_468469_object127:head# 2024-04-25T09:33:56.629 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:39249c23:::benchmark_data_smithi134_462957_object161:head# 2024-04-25T09:33:56.630 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3a8ee0cf:::benchmark_data_smithi134_462957_object151:head# 2024-04-25T09:33:56.630 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3b3824a5:::benchmark_data_smithi134_468469_object111:head# 2024-04-25T09:33:56.631 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3b9ee75a:::benchmark_data_smithi134_462957_object326:head# 2024-04-25T09:33:56.632 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3d341765:::benchmark_data_smithi134_468469_object62:head# 2024-04-25T09:33:56.632 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3e5f4585:::benchmark_data_smithi134_462957_object47:head# 2024-04-25T09:33:56.633 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3ed5c4c3:::benchmark_data_smithi134_462957_object205:head# 2024-04-25T09:33:56.634 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3f3910d9:::benchmark_data_smithi134_468469_object98:head# 2024-04-25T09:33:56.634 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3fc6ca95:::benchmark_data_smithi134_462957_object12:head# 2024-04-25T09:33:56.635 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:40470fd1:::benchmark_data_smithi134_462957_object17:head# 2024-04-25T09:33:56.635 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:417961ce:::benchmark_data_smithi134_462957_object319:head# 2024-04-25T09:33:56.636 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:418ec0ee:::benchmark_data_smithi134_462957_object149:head# 2024-04-25T09:33:56.637 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:419edc7e:::benchmark_data_smithi134_468469_object26:head# 2024-04-25T09:33:56.638 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:422d92f6:::benchmark_data_smithi134_462957_object272:head# 2024-04-25T09:33:56.638 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:42b97c1d:::benchmark_data_smithi134_468469_object112:head# 2024-04-25T09:33:56.639 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:42c245f2:::benchmark_data_smithi134_468469_object99:head# 2024-04-25T09:33:56.639 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:435e2086:::benchmark_data_smithi134_462957_object171:head# 2024-04-25T09:33:56.640 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:441b2425:::benchmark_data_smithi134_468469_object136:head# 2024-04-25T09:33:56.641 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:444a49fd:::benchmark_data_smithi134_462957_object328:head# 2024-04-25T09:33:56.641 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:44a8dd41:::benchmark_data_smithi134_462957_object3:head# 2024-04-25T09:33:56.642 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:467046b5:::benchmark_data_smithi134_462957_object265:head# 2024-04-25T09:33:56.643 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4672b82d:::benchmark_data_smithi134_462957_object295:head# 2024-04-25T09:33:56.643 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:47196e71:::benchmark_data_smithi134_462957_object269:head# 2024-04-25T09:33:56.644 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:478e238f:::benchmark_data_smithi134_462957_object213:head# 2024-04-25T09:33:56.644 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:47a3ca01:::benchmark_data_smithi134_468469_object90:head# 2024-04-25T09:33:56.646 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:47e25b7f:::benchmark_data_smithi134_462957_object249:head# 2024-04-25T09:33:56.646 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:47f5dd1f:::benchmark_data_smithi134_462957_object122:head# 2024-04-25T09:33:56.647 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:49a59676:::benchmark_data_smithi134_468469_object4:head# 2024-04-25T09:33:56.648 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:49e180d1:::benchmark_data_smithi134_462957_object88:head# 2024-04-25T09:33:56.648 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4a676de3:::benchmark_data_smithi134_462957_object263:head# 2024-04-25T09:33:56.649 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4a88808c:::benchmark_data_smithi134_468469_object80:head# 2024-04-25T09:33:56.650 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4b79b182:::benchmark_data_smithi134_462957_object137:head# 2024-04-25T09:33:56.650 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4c691a21:::benchmark_data_smithi134_468469_object89:head# 2024-04-25T09:33:56.651 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4cb6f1b7:::benchmark_data_smithi134_462957_object78:head# 2024-04-25T09:33:56.652 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4e62c3c1:::benchmark_data_smithi134_462957_object53:head# 2024-04-25T09:33:56.652 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4e981d63:::benchmark_data_smithi134_462957_object227:head# 2024-04-25T09:33:56.653 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4fda476d:::benchmark_data_smithi134_462957_object194:head# 2024-04-25T09:33:56.653 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:504c6844:::benchmark_data_smithi134_462957_object94:head# 2024-04-25T09:33:56.654 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:504ee13c:::benchmark_data_smithi134_462957_object93:head# 2024-04-25T09:33:56.655 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:510547ae:::benchmark_data_smithi134_468469_object46:head# 2024-04-25T09:33:56.656 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:511d8705:::benchmark_data_smithi134_462957_object271:head# 2024-04-25T09:33:56.657 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:51b089df:::benchmark_data_smithi134_462957_object273:head# 2024-04-25T09:33:56.657 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5266c200:::benchmark_data_smithi134_462957_object32:head# 2024-04-25T09:33:56.658 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:542d36c1:::benchmark_data_smithi134_462957_object4:head# 2024-04-25T09:33:56.659 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:544671af:::benchmark_data_smithi134_462957_object172:head# 2024-04-25T09:33:56.659 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:549eba21:::benchmark_data_smithi134_462957_object133:head# 2024-04-25T09:33:56.660 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:556d8a79:::benchmark_data_smithi134_462957_object214:head# 2024-04-25T09:33:56.660 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:556e8b3c:::benchmark_data_smithi134_462957_object96:head# 2024-04-25T09:33:56.661 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:55950821:::benchmark_data_smithi134_462957_object141:head# 2024-04-25T09:33:56.661 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:55fef97d:::benchmark_data_smithi134_468469_object38:head# 2024-04-25T09:33:56.662 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:560a6861:::benchmark_data_smithi134_468469_object131:head# 2024-04-25T09:33:56.663 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:575ccb6b:::benchmark_data_smithi134_462957_object280:head# 2024-04-25T09:33:56.663 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:57a020db:::benchmark_data_smithi134_462957_object286:head# 2024-04-25T09:33:56.664 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5853733d:::benchmark_data_smithi134_468469_object119:head# 2024-04-25T09:33:56.665 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:58587487:::benchmark_data_smithi134_462957_object51:head# 2024-04-25T09:33:56.665 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5b1ef10f:::benchmark_data_smithi134_468469_object132:head# 2024-04-25T09:33:56.666 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5b32c1cc:::benchmark_data_smithi134_468469_object7:head# 2024-04-25T09:33:56.666 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5c373486:::benchmark_data_smithi134_462957_object14:head# 2024-04-25T09:33:56.667 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5d0c53ae:::benchmark_data_smithi134_468469_object123:head# 2024-04-25T09:33:56.668 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5d23a64b:::benchmark_data_smithi134_468469_object133:head# 2024-04-25T09:33:56.668 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5dada283:::benchmark_data_smithi134_462957_object77:head# 2024-04-25T09:33:56.669 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5e2dac6b:::benchmark_data_smithi134_468469_object81:head# 2024-04-25T09:33:56.670 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5e48ae44:::benchmark_data_smithi134_462957_object333:head# 2024-04-25T09:33:56.671 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5e9f3007:::benchmark_data_smithi134_468469_object1:head# 2024-04-25T09:33:56.671 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6066d321:::benchmark_data_smithi134_468469_object126:head# 2024-04-25T09:33:56.672 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:60d506d0:::benchmark_data_smithi134_462957_object111:head# 2024-04-25T09:33:56.673 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:610e8400:::benchmark_data_smithi134_468469_object97:head# 2024-04-25T09:33:56.673 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:61342236:::benchmark_data_smithi134_462957_object301:head# 2024-04-25T09:33:56.674 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:61c961a6:::benchmark_data_smithi134_468469_object61:head# 2024-04-25T09:33:56.674 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:61d1b2a4:::benchmark_data_smithi134_462957_object179:head# 2024-04-25T09:33:56.675 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:621d43c5:::benchmark_data_smithi134_462957_object102:head# 2024-04-25T09:33:56.676 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:62545d41:::benchmark_data_smithi134_462957_object318:head# 2024-04-25T09:33:56.676 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:632309d1:::benchmark_data_smithi134_468469_object0:head# 2024-04-25T09:33:56.677 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:632b2ff2:::benchmark_data_smithi134_462957_object113:head# 2024-04-25T09:33:56.678 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:63effe0b:::benchmark_data_smithi134_462957_object1:head# 2024-04-25T09:33:56.678 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:63f38e63:::benchmark_data_smithi134_462957_object314:head# 2024-04-25T09:33:56.679 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:645caf70:::benchmark_data_smithi134_462957_object9:head# 2024-04-25T09:33:56.680 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:64cb656a:::benchmark_data_smithi134_462957_object38:head# 2024-04-25T09:33:56.680 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:651d1c11:::benchmark_data_smithi134_462957_object219:head# 2024-04-25T09:33:56.681 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:65698cf1:::benchmark_data_smithi134_462957_object315:head# 2024-04-25T09:33:56.681 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:660b1875:::benchmark_data_smithi134_462957_object153:head# 2024-04-25T09:33:56.681 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6631758f:::benchmark_data_smithi134_468469_object66:head# 2024-04-25T09:33:56.683 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:66869d35:::benchmark_data_smithi134_468469_object102:head# 2024-04-25T09:33:56.684 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:66e4bb20:::benchmark_data_smithi134_462957_object259:head# 2024-04-25T09:33:56.684 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:670b93a5:::benchmark_data_smithi134_468469_object85:head# 2024-04-25T09:33:56.685 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:67406b53:::benchmark_data_smithi134_462957_object168:head# 2024-04-25T09:33:56.685 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6753f201:::benchmark_data_smithi134_462957_object19:head# 2024-04-25T09:33:56.686 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:67de4446:::benchmark_data_smithi134_462957_object11:head# 2024-04-25T09:33:56.687 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68078f8b:::benchmark_data_smithi134_468469_object129:head# 2024-04-25T09:33:56.687 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68431e33:::benchmark_data_smithi134_462957_object6:head# 2024-04-25T09:33:56.688 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68da1af8:::benchmark_data_smithi134_462957_object224:head# 2024-04-25T09:33:56.689 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68e29e1b:::benchmark_data_smithi134_462957_object223:head# 2024-04-25T09:33:56.689 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:68ff3868:::benchmark_data_smithi134_462957_object104:head# 2024-04-25T09:33:56.690 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:69444b58:::benchmark_data_smithi134_462957_object236:head# 2024-04-25T09:33:56.690 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:69ce30ac:::benchmark_data_smithi134_468469_object128:head# 2024-04-25T09:33:56.692 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6b5b0c73:::benchmark_data_smithi134_468469_object47:head# 2024-04-25T09:33:56.692 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6bef445e:::benchmark_data_smithi134_462957_object145:head# 2024-04-25T09:33:56.693 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6c6b0a68:::benchmark_data_smithi134_462957_object106:head# 2024-04-25T09:33:56.693 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6cb36ab9:::benchmark_data_smithi134_462957_object128:head# 2024-04-25T09:33:56.694 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6cb830fe:::benchmark_data_smithi134_462957_object281:head# 2024-04-25T09:33:56.695 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6d038ea7:::benchmark_data_smithi134_462957_object250:head# 2024-04-25T09:33:56.696 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6dfad7ef:::benchmark_data_smithi134_462957_object253:head# 2024-04-25T09:33:56.696 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6e8445ed:::benchmark_data_smithi134_462957_object119:head# 2024-04-25T09:33:56.696 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6ea927e3:::benchmark_data_smithi134_462957_object67:head# 2024-04-25T09:33:56.697 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6edbe582:::benchmark_data_smithi134_462957_object260:head# 2024-04-25T09:33:56.697 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6f02c381:::benchmark_data_smithi134_468469_object27:head# 2024-04-25T09:33:56.698 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6f1d965d:::benchmark_data_smithi134_468469_object101:head# 2024-04-25T09:33:56.699 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6f922ade:::benchmark_data_smithi134_462957_object136:head# 2024-04-25T09:33:56.699 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6fe808b0:::benchmark_data_smithi134_462957_object7:head# 2024-04-25T09:33:56.700 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:71c24ec6:::benchmark_data_smithi134_462957_object35:head# 2024-04-25T09:33:56.700 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7280df13:::benchmark_data_smithi134_462957_object109:head# 2024-04-25T09:33:56.701 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:72b74ac7:::benchmark_data_smithi134_462957_object292:head# 2024-04-25T09:33:56.702 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:72d020e0:::benchmark_data_smithi134_462957_object20:head# 2024-04-25T09:33:56.702 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:73570709:::benchmark_data_smithi134_462957_object244:head# 2024-04-25T09:33:56.703 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:73fa9d47:::benchmark_data_smithi134_462957_object0:head# 2024-04-25T09:33:56.703 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:76387380:::benchmark_data_smithi134_462957_object103:head# 2024-04-25T09:33:56.703 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:77414e26:::benchmark_data_smithi134_468469_object144:head# 2024-04-25T09:33:56.704 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:77a61cee:::benchmark_data_smithi134_462957_object237:head# 2024-04-25T09:33:56.705 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7826dd0b:::benchmark_data_smithi134_462957_object233:head# 2024-04-25T09:33:56.706 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:787a592c:::benchmark_data_smithi134_468469_object108:head# 2024-04-25T09:33:56.706 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7958a491:::benchmark_data_smithi134_462957_object232:head# 2024-04-25T09:33:56.707 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7a5eed6c:::benchmark_data_smithi134_462957_object55:head# 2024-04-25T09:33:56.707 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7a922587:::benchmark_data_smithi134_468469_object39:head# 2024-04-25T09:33:56.708 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7bed12e2:::benchmark_data_smithi134_462957_object90:head# 2024-04-25T09:33:56.709 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7c53f004:::benchmark_data_smithi134_462957_object183:head# 2024-04-25T09:33:56.709 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7cc751e9:::benchmark_data_smithi134_462957_object150:head# 2024-04-25T09:33:56.710 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7de067cb:::benchmark_data_smithi134_462957_object148:head# 2024-04-25T09:33:56.711 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7e10a619:::benchmark_data_smithi134_462957_object296:head# 2024-04-25T09:33:56.712 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7f2f5709:::benchmark_data_smithi134_462957_object58:head# 2024-04-25T09:33:56.712 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7f4d9c89:::benchmark_data_smithi134_462957_object209:head# 2024-04-25T09:33:56.712 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7fa16e9a:::benchmark_data_smithi134_468469_object51:head# 2024-04-25T09:33:56.713 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7fa509a6:::benchmark_data_smithi134_468469_object120:head# 2024-04-25T09:33:56.714 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:80b4c8b2:::benchmark_data_smithi134_468469_object22:head# 2024-04-25T09:33:56.715 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:80f5f423:::benchmark_data_smithi134_462957_object61:head# 2024-04-25T09:33:56.722 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8136ddfc:::benchmark_data_smithi134_462957_object181:head# 2024-04-25T09:33:56.723 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:820a3e6e:::benchmark_data_smithi134_468469_object24:head# 2024-04-25T09:33:56.723 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:822e7d0d:::benchmark_data_smithi134_462957_object36:head# 2024-04-25T09:33:56.724 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:82344c27:::benchmark_data_smithi134_462957_object239:head# 2024-04-25T09:33:56.724 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:82ab8645:::benchmark_data_smithi134_462957_object312:head# 2024-04-25T09:33:56.725 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:83241f30:::benchmark_data_smithi134_462957_object186:head# 2024-04-25T09:33:56.726 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:833e47e2:::benchmark_data_smithi134_462957_object201:head# 2024-04-25T09:33:56.727 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:83476594:::benchmark_data_smithi134_468469_object78:head# 2024-04-25T09:33:56.727 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:838de8c2:::benchmark_data_smithi134_468469_object121:head# 2024-04-25T09:33:56.728 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:83909346:::benchmark_data_smithi134_462957_object165:head# 2024-04-25T09:33:56.729 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8486115e:::benchmark_data_smithi134_462957_object134:head# 2024-04-25T09:33:56.729 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:85663fce:::benchmark_data_smithi134_468469_object104:head# 2024-04-25T09:33:56.730 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:85d49e01:::benchmark_data_smithi134_462957_object114:head# 2024-04-25T09:33:56.731 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:86e69dbb:::benchmark_data_smithi134_462957_object334:head# 2024-04-25T09:33:56.731 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:87789c1f:::benchmark_data_smithi134_462957_object299:head# 2024-04-25T09:33:56.732 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:87e9176e:::benchmark_data_smithi134_462957_object176:head# 2024-04-25T09:33:56.732 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:884c4353:::benchmark_data_smithi134_468469_object86:head# 2024-04-25T09:33:56.733 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:88b81233:::benchmark_data_smithi134_462957_object197:head# 2024-04-25T09:33:56.734 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8a093cc3:::benchmark_data_smithi134_468469_object14:head# 2024-04-25T09:33:56.734 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8a1b1ae0:::benchmark_data_smithi134_468469_object91:head# 2024-04-25T09:33:56.735 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8a55344d:::benchmark_data_smithi134_462957_object163:head# 2024-04-25T09:33:56.736 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8b35579f:::benchmark_data_smithi134_462957_object101:head# 2024-04-25T09:33:56.736 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8c42d79e:::benchmark_data_smithi134_462957_object285:head# 2024-04-25T09:33:56.737 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8c5cb74b:::benchmark_data_smithi134_462957_object30:head# 2024-04-25T09:33:56.737 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8c777440:::benchmark_data_smithi134_462957_object69:head# 2024-04-25T09:33:56.738 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8d18392e:::benchmark_data_smithi134_462957_object42:head# 2024-04-25T09:33:56.738 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8d4a51df:::benchmark_data_smithi134_462957_object202:head# 2024-04-25T09:33:56.739 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8f9fc238:::benchmark_data_smithi134_462957_object222:head# 2024-04-25T09:33:56.739 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8fcd40aa:::benchmark_data_smithi134_462957_object13:head# 2024-04-25T09:33:56.739 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8ffd53b9:::benchmark_data_smithi134_462957_object152:head# 2024-04-25T09:33:56.740 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9102906f:::benchmark_data_smithi134_468469_object115:head# 2024-04-25T09:33:56.741 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:912ceabf:::benchmark_data_smithi134_462957_object291:head# 2024-04-25T09:33:56.741 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:913e15cc:::benchmark_data_smithi134_462957_object283:head# 2024-04-25T09:33:56.742 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:92bf16d1:::benchmark_data_smithi134_462957_object195:head# 2024-04-25T09:33:56.743 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:92ed1f0f:::benchmark_data_smithi134_462957_object65:head# 2024-04-25T09:33:56.743 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:93c080bd:::benchmark_data_smithi134_468469_object76:head# 2024-04-25T09:33:56.744 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9417108d:::benchmark_data_smithi134_462957_object261:head# 2024-04-25T09:33:56.745 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:94a46faa:::benchmark_data_smithi134_468469_object32:head# 2024-04-25T09:33:56.745 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:954cc32c:::benchmark_data_smithi134_462957_object139:head# 2024-04-25T09:33:56.746 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9632b147:::benchmark_data_smithi134_462957_object154:head# 2024-04-25T09:33:56.746 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:96994074:::benchmark_data_smithi134_468469_object5:head# 2024-04-25T09:33:56.747 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9726b3f5:::benchmark_data_smithi134_462957_object135:head# 2024-04-25T09:33:56.748 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:97aa80c9:::benchmark_data_smithi134_462957_object210:head# 2024-04-25T09:33:56.748 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:98039778:::benchmark_data_smithi134_468469_object110:head# 2024-04-25T09:33:56.748 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9870b61d:::benchmark_data_smithi134_462957_object50:head# 2024-04-25T09:33:56.749 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9a3c902a:::benchmark_data_smithi134_462957_object66:head# 2024-04-25T09:33:56.749 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9a63a41b:::benchmark_data_smithi134_462957_object59:head# 2024-04-25T09:33:56.750 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9a6d1805:::benchmark_data_smithi134_468469_object141:head# 2024-04-25T09:33:56.750 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9af4c21c:::benchmark_data_smithi134_462957_object99:head# 2024-04-25T09:33:56.751 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9c02c888:::benchmark_data_smithi134_462957_object100:head# 2024-04-25T09:33:56.751 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9c7daede:::benchmark_data_smithi134_468469_object71:head# 2024-04-25T09:33:56.752 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9d244c8f:::benchmark_data_smithi134_468469_object18:head# 2024-04-25T09:33:56.753 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9d740ac7:::benchmark_data_smithi134_462957_object156:head# 2024-04-25T09:33:56.753 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9d9951df:::benchmark_data_smithi134_468469_object9:head# 2024-04-25T09:33:56.753 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9e32be9a:::benchmark_data_smithi134_462957_object54:head# 2024-04-25T09:33:56.754 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9e6056cd:::benchmark_data_smithi134_462957_object164:head# 2024-04-25T09:33:56.755 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9e878f34:::benchmark_data_smithi134_468469_object54:head# 2024-04-25T09:33:56.755 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9ebf06db:::benchmark_data_smithi134_462957_object284:head# 2024-04-25T09:33:56.755 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9ee7691c:::benchmark_data_smithi134_468469_object65:head# 2024-04-25T09:33:56.756 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a045e8d8:::benchmark_data_smithi134_462957_object234:head# 2024-04-25T09:33:56.756 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a08b0a4d:::benchmark_data_smithi134_462957_object241:head# 2024-04-25T09:33:56.757 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a0c022f8:::benchmark_data_smithi134_462957_object226:head# 2024-04-25T09:33:56.757 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a1658627:::benchmark_data_smithi134_462957_object327:head# 2024-04-25T09:33:56.758 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a1bb4fbf:::benchmark_data_smithi134_462957_object198:head# 2024-04-25T09:33:56.760 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a2b4a5bd:::benchmark_data_smithi134_462957_object27:head# 2024-04-25T09:33:56.760 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a3329f50:::benchmark_data_smithi134_468469_object107:head# 2024-04-25T09:33:56.761 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a50fb507:::benchmark_data_smithi134_468469_object118:head# 2024-04-25T09:33:56.761 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a5409620:::benchmark_data_smithi134_468469_object60:head# 2024-04-25T09:33:56.762 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a5ca7c00:::benchmark_data_smithi134_462957_object72:head# 2024-04-25T09:33:56.762 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a5f62c63:::benchmark_data_smithi134_462957_object316:head# 2024-04-25T09:33:56.763 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a63f834f:::benchmark_data_smithi134_462957_object157:head# 2024-04-25T09:33:56.763 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a715009b:::benchmark_data_smithi134_462957_object169:head# 2024-04-25T09:33:56.764 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a85dbb56:::benchmark_data_smithi134_462957_object307:head# 2024-04-25T09:33:56.764 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a8767535:::benchmark_data_smithi134_468469_object92:head# 2024-04-25T09:33:56.765 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a8d1cd5f:::benchmark_data_smithi134_462957_object107:head# 2024-04-25T09:33:56.766 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a9621e84:::benchmark_data_smithi134_462957_object303:head# 2024-04-25T09:33:56.766 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a9a511c7:::benchmark_data_smithi134_462957_object321:head# 2024-04-25T09:33:56.767 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a9a8cefb:::benchmark_data_smithi134_468469_object36:head# 2024-04-25T09:33:56.767 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:aa4d3c89:::benchmark_data_smithi134_468469_object34:head# 2024-04-25T09:33:56.768 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ab3c651b:::benchmark_data_smithi134_468469_object58:head# 2024-04-25T09:33:56.768 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ab6edda1:::benchmark_data_smithi134_462957_object212:head# 2024-04-25T09:33:56.769 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:aba338b0:::benchmark_data_smithi134_462957_object184:head# 2024-04-25T09:33:56.769 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:abdd83aa:::benchmark_data_smithi134_462957_object85:head# 2024-04-25T09:33:56.770 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:abf2d6a2:::benchmark_data_smithi134_468469_object48:head# 2024-04-25T09:33:56.770 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ac85576b:::benchmark_data_smithi134_462957_object282:head# 2024-04-25T09:33:56.771 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ad4a0ace:::benchmark_data_smithi134_468469_object116:head# 2024-04-25T09:33:56.771 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ae25ce97:::benchmark_data_smithi134_462957_object75:head# 2024-04-25T09:33:56.772 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:aeaa246a:::benchmark_data_smithi134_468469_object88:head# 2024-04-25T09:33:56.772 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b1f8b3b5:::benchmark_data_smithi134_462957_object160:head# 2024-04-25T09:33:56.773 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b249419b:::benchmark_data_smithi134_462957_object275:head# 2024-04-25T09:33:56.773 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b289940b:::benchmark_data_smithi134_462957_object26:head# 2024-04-25T09:33:56.774 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b29d1223:::benchmark_data_smithi134_462957_object167:head# 2024-04-25T09:33:56.774 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b4ae5688:::benchmark_data_smithi134_462957_object182:head# 2024-04-25T09:33:56.775 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b5565581:::benchmark_data_smithi134_462957_object22:head# 2024-04-25T09:33:56.776 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b56627a7:::benchmark_data_smithi134_462957_object108:head# 2024-04-25T09:33:56.776 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b62c98b7:::benchmark_data_smithi134_468469_object137:head# 2024-04-25T09:33:56.777 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b6ff86cd:::benchmark_data_smithi134_462957_object166:head# 2024-04-25T09:33:56.777 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b7edf419:::benchmark_data_smithi134_462957_object84:head# 2024-04-25T09:33:56.778 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:33:56.778 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b8f154ff:::benchmark_data_smithi134_462957_object206:head# 2024-04-25T09:33:56.779 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b920d5ca:::benchmark_data_smithi134_468469_object94:head# 2024-04-25T09:33:56.780 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b93070b3:::benchmark_data_smithi134_468469_object55:head# 2024-04-25T09:33:56.780 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b93b7738:::benchmark_data_smithi134_462957_object92:head# 2024-04-25T09:33:56.781 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b997874c:::benchmark_data_smithi134_462957_object320:head# 2024-04-25T09:33:56.781 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ba34c80e:::benchmark_data_smithi134_462957_object29:head# 2024-04-25T09:33:56.782 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ba9726b3:::benchmark_data_smithi134_468469_object95:head# 2024-04-25T09:33:56.783 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bb51ab09:::benchmark_data_smithi134_462957_object246:head# 2024-04-25T09:33:56.783 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bb9ed292:::benchmark_data_smithi134_462957_object44:head# 2024-04-25T09:33:56.784 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bc0fe2eb:::benchmark_data_smithi134_462957_object105:head# 2024-04-25T09:33:56.785 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bd33c13c:::benchmark_data_smithi134_462957_object121:head# 2024-04-25T09:33:56.785 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bd52ec16:::benchmark_data_smithi134_462957_object68:head# 2024-04-25T09:33:56.786 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bd71e327:::benchmark_data_smithi134_468469_object10:head# 2024-04-25T09:33:56.786 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:be19e778:::benchmark_data_smithi134_462957_object238:head# 2024-04-25T09:33:56.787 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bf64a100:::benchmark_data_smithi134_462957_object83:head# 2024-04-25T09:33:56.788 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c03f464f:::benchmark_data_smithi134_462957_object18:head# 2024-04-25T09:33:56.788 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c0ec0650:::benchmark_data_smithi134_468469_object93:head# 2024-04-25T09:33:56.789 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c0f283ce:::benchmark_data_smithi134_462957_object124:head# 2024-04-25T09:33:56.790 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c16c7b78:::benchmark_data_smithi134_462957_object2:head# 2024-04-25T09:33:56.790 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c184615c:::benchmark_data_smithi134_468469_object28:head# 2024-04-25T09:33:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c18f6438:::benchmark_data_smithi134_462957_object220:head# 2024-04-25T09:33:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c244846a:::benchmark_data_smithi134_462957_object329:head# 2024-04-25T09:33:56.792 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c274bcc9:::benchmark_data_smithi134_462957_object332:head# 2024-04-25T09:33:56.793 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c2e9f6b4:::benchmark_data_smithi134_462957_object330:head# 2024-04-25T09:33:56.793 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c35cfddf:::benchmark_data_smithi134_462957_object200:head# 2024-04-25T09:33:56.794 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c35d1837:::benchmark_data_smithi134_468469_object103:head# 2024-04-25T09:33:56.795 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c47b78d0:::benchmark_data_smithi134_462957_object117:head# 2024-04-25T09:33:56.795 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c6365174:::benchmark_data_smithi134_468469_object21:head# 2024-04-25T09:33:56.796 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c70a7496:::benchmark_data_smithi134_462957_object60:head# 2024-04-25T09:33:56.796 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c73308dd:::benchmark_data_smithi134_462957_object258:head# 2024-04-25T09:33:56.797 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c74b145d:::benchmark_data_smithi134_462957_object146:head# 2024-04-25T09:33:56.798 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c7d37ef1:::benchmark_data_smithi134_462957_object125:head# 2024-04-25T09:33:56.798 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c7e782f1:::benchmark_data_smithi134_468469_object84:head# 2024-04-25T09:33:56.799 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c84d6c63:::benchmark_data_smithi134_462957_object221:head# 2024-04-25T09:33:56.800 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c857d305:::benchmark_data_smithi134_468469_object96:head# 2024-04-25T09:33:56.800 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c8d637f7:::benchmark_data_smithi134_462957_object306:head# 2024-04-25T09:33:56.801 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c9e347b1:::benchmark_data_smithi134_462957_object242:head# 2024-04-25T09:33:56.802 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ca1ed64b:::benchmark_data_smithi134_468469_object135:head# 2024-04-25T09:33:56.802 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ca2673ae:::benchmark_data_smithi134_468469_object125:head# 2024-04-25T09:33:56.803 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ca46b863:::benchmark_data_smithi134_462957_object57:head# 2024-04-25T09:33:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ca795a74:::benchmark_data_smithi134_462957_object81:head# 2024-04-25T09:33:56.804 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cae41921:::benchmark_data_smithi134_468469_object106:head# 2024-04-25T09:33:56.805 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cc49d25d:::benchmark_data_smithi134_462957_object144:head# 2024-04-25T09:33:56.806 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cc623ab1:::benchmark_data_smithi134_468469_object25:head# 2024-04-25T09:33:56.806 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ccd8cbfe:::benchmark_data_smithi134_462957_object5:head# 2024-04-25T09:33:56.807 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cce05002:::benchmark_data_smithi134_462957_object130:head# 2024-04-25T09:33:56.807 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ccfcf8fa:::benchmark_data_smithi134_468469_object70:head# 2024-04-25T09:33:56.808 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cd3e8f8e:::benchmark_data_smithi134_468469_object53:head# 2024-04-25T09:33:56.809 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cd79c5f8:::benchmark_data_smithi134_462957_object76:head# 2024-04-25T09:33:56.809 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cdb281d5:::benchmark_data_smithi134_462957_object46:head# 2024-04-25T09:33:56.810 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cdc69e54:::benchmark_data_smithi134_462957_object252:head# 2024-04-25T09:33:56.810 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cf2a4062:::benchmark_data_smithi134_462957_object199:head# 2024-04-25T09:33:56.811 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cf35812c:::benchmark_data_smithi134_462957_object243:head# 2024-04-25T09:33:56.812 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cf693ea1:::benchmark_data_smithi134_462957_object147:head# 2024-04-25T09:33:56.812 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d0dad50c:::benchmark_data_smithi134_462957_object97:head# 2024-04-25T09:33:56.812 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d1761cb1:::benchmark_data_smithi134_462957_object251:head# 2024-04-25T09:33:56.813 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d1b7fc50:::benchmark_data_smithi134_462957_object211:head# 2024-04-25T09:33:56.813 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d2bd3105:::benchmark_data_smithi134_462957_object277:head# 2024-04-25T09:33:56.814 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d3201ed2:::benchmark_data_smithi134_462957_object45:head# 2024-04-25T09:33:56.814 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d3f96c8d:::benchmark_data_smithi134_468469_object140:head# 2024-04-25T09:33:56.815 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d423debd:::benchmark_data_smithi134_462957_object115:head# 2024-04-25T09:33:56.816 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d4924696:::benchmark_data_smithi134_468469_object35:head# 2024-04-25T09:33:56.816 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d4c4f287:::benchmark_data_smithi134_462957_object21:head# 2024-04-25T09:33:56.817 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d50a78a1:::benchmark_data_smithi134_462957_object25:head# 2024-04-25T09:33:56.818 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d5af3bd1:::benchmark_data_smithi134_462957_object180:head# 2024-04-25T09:33:56.818 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d5ef9c20:::benchmark_data_smithi134_462957_object70:head# 2024-04-25T09:33:56.819 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d6221551:::benchmark_data_smithi134_468469_object74:head# 2024-04-25T09:33:56.819 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d65dc781:::benchmark_data_smithi134_462957_object323:head# 2024-04-25T09:33:56.820 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d6872c2f:::benchmark_data_smithi134_462957_object192:head# 2024-04-25T09:33:56.820 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d6cbb661:::benchmark_data_smithi134_462957_object279:head# 2024-04-25T09:33:56.821 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d6d8a7de:::benchmark_data_smithi134_462957_object175:head# 2024-04-25T09:33:56.822 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d9df41a0:::benchmark_data_smithi134_462957_object288:head# 2024-04-25T09:33:56.822 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dba2aa95:::benchmark_data_smithi134_462957_object31:head# 2024-04-25T09:33:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dc540d1b:::benchmark_data_smithi134_462957_object189:head# 2024-04-25T09:33:56.823 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dc6e92dd:::benchmark_data_smithi134_462957_object159:head# 2024-04-25T09:33:56.824 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dc96f6c8:::benchmark_data_smithi134_468469_object3:head# 2024-04-25T09:33:56.824 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dcca6922:::benchmark_data_smithi134_462957_object311:head# 2024-04-25T09:33:56.825 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd1204a0:::benchmark_data_smithi134_462957_object71:head# 2024-04-25T09:33:56.826 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd2acaa8:::benchmark_data_smithi134_468469_object2:head# 2024-04-25T09:33:56.826 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd3e6d82:::benchmark_data_smithi134_462957_object262:head# 2024-04-25T09:33:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:df8a97e3:::benchmark_data_smithi134_468469_object33:head# 2024-04-25T09:33:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dfc1c8cf:::benchmark_data_smithi134_462957_object229:head# 2024-04-25T09:33:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e04321de:::benchmark_data_smithi134_468469_object114:head# 2024-04-25T09:33:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e06b8840:::benchmark_data_smithi134_462957_object91:head# 2024-04-25T09:33:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e06cab0a:::benchmark_data_smithi134_462957_object23:head# 2024-04-25T09:33:56.831 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e2819600:::benchmark_data_smithi134_468469_object12:head# 2024-04-25T09:33:56.831 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e28f6f75:::benchmark_data_smithi134_462957_object300:head# 2024-04-25T09:33:56.831 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e2be0804:::benchmark_data_smithi134_462957_object235:head# 2024-04-25T09:33:56.832 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e2c8c0bf:::benchmark_data_smithi134_462957_object40:head# 2024-04-25T09:33:56.833 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e36a9ada:::benchmark_data_smithi134_468469_object100:head# 2024-04-25T09:33:56.833 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e3a40c39:::benchmark_data_smithi134_462957_object63:head# 2024-04-25T09:33:56.834 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e3c5ddbe:::benchmark_data_smithi134_468469_object64:head# 2024-04-25T09:33:56.835 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e5228945:::benchmark_data_smithi134_462957_object293:head# 2024-04-25T09:33:56.835 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e5b32a73:::benchmark_data_smithi134_468469_object37:head# 2024-04-25T09:33:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e614399c:::benchmark_data_smithi134_462957_object110:head# 2024-04-25T09:33:56.836 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e65dd3c4:::benchmark_data_smithi134_468469_object130:head# 2024-04-25T09:33:56.837 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e65f056d:::benchmark_data_smithi134_462957_object196:head# 2024-04-25T09:33:56.838 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e6a6c722:::benchmark_data_smithi134_462957_object264:head# 2024-04-25T09:33:56.838 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e74bd302:::benchmark_data_smithi134_462957_object131:head# 2024-04-25T09:33:56.839 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e795db49:::benchmark_data_smithi134_468469_object42:head# 2024-04-25T09:33:56.839 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e7aa26a1:::benchmark_data_smithi134_462957_object129:head# 2024-04-25T09:33:56.840 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e8378195:::benchmark_data_smithi134_462957_object256:head# 2024-04-25T09:33:56.841 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e8ce525c:::benchmark_data_smithi134_462957_object155:head# 2024-04-25T09:33:56.841 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e8f5abd5:::benchmark_data_smithi134_462957_object56:head# 2024-04-25T09:33:56.842 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e93b4e82:::benchmark_data_smithi134_468469_object16:head# 2024-04-25T09:33:56.843 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e985cf7e:::benchmark_data_smithi134_462957_object287:head# 2024-04-25T09:33:56.843 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e9f08f42:::benchmark_data_smithi134_462957_object187:head# 2024-04-25T09:33:56.844 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ea484b6f:::benchmark_data_smithi134_462957_object82:head# 2024-04-25T09:33:56.844 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:eb7589e9:::benchmark_data_smithi134_468469_object59:head# 2024-04-25T09:33:56.845 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ed2fe309:::benchmark_data_smithi134_468469_object134:head# 2024-04-25T09:33:56.846 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ee73ec6d:::benchmark_data_smithi134_462957_object33:head# 2024-04-25T09:33:56.846 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ee90a495:::benchmark_data_smithi134_468469_object57:head# 2024-04-25T09:33:56.847 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ef28e1d3:::benchmark_data_smithi134_468469_object31:head# 2024-04-25T09:33:56.847 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f01022da:::benchmark_data_smithi134_462957_object324:head# 2024-04-25T09:33:56.848 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f030e4b2:::benchmark_data_smithi134_468469_object67:head# 2024-04-25T09:33:56.848 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f0800d50:::benchmark_data_smithi134_462957_object217:head# 2024-04-25T09:33:56.849 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f0c0db8e:::benchmark_data_smithi134_468469_object19:head# 2024-04-25T09:33:56.850 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f0f00dfc:::benchmark_data_smithi134_462957_object185:head# 2024-04-25T09:33:56.850 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f0f2cea2:::benchmark_data_smithi134_462957_object317:head# 2024-04-25T09:33:56.851 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f131ea49:::benchmark_data_smithi134_462957_object15:head# 2024-04-25T09:33:56.851 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f24c3070:::benchmark_data_smithi134_468469_object79:head# 2024-04-25T09:33:56.852 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f3c72ea5:::benchmark_data_smithi134_462957_object173:head# 2024-04-25T09:33:56.853 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f439d218:::benchmark_data_smithi134_468469_object30:head# 2024-04-25T09:33:56.853 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f581c1b4:::benchmark_data_smithi134_462957_object218:head# 2024-04-25T09:33:56.854 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f63f8b4d:::benchmark_data_smithi134_462957_object247:head# 2024-04-25T09:33:56.855 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f688bb37:::benchmark_data_smithi134_468469_object139:head# 2024-04-25T09:33:56.855 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f6fa688e:::benchmark_data_smithi134_468469_object142:head# 2024-04-25T09:33:56.856 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f70159a7:::benchmark_data_smithi134_462957_object208:head# 2024-04-25T09:33:56.856 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f741ec8b:::benchmark_data_smithi134_462957_object325:head# 2024-04-25T09:33:56.857 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f7f2b0bd:::benchmark_data_smithi134_462957_object302:head# 2024-04-25T09:33:56.858 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f849895e:::benchmark_data_smithi134_468469_object11:head# 2024-04-25T09:33:56.858 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f8a62bfb:::benchmark_data_smithi134_468469_object122:head# 2024-04-25T09:33:56.859 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f8d2618b:::benchmark_data_smithi134_462957_object255:head# 2024-04-25T09:33:56.859 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f8dab886:::benchmark_data_smithi134_468469_object82:head# 2024-04-25T09:33:56.860 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f8f97fba:::benchmark_data_smithi134_462957_object178:head# 2024-04-25T09:33:56.861 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f9f3f45e:::benchmark_data_smithi134_468469_object143:head# 2024-04-25T09:33:56.861 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fa5b5d21:::benchmark_data_smithi134_462957_object274:head# 2024-04-25T09:33:56.862 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fa93c7ab:::benchmark_data_smithi134_468469_object63:head# 2024-04-25T09:33:56.863 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:faa372e4:::benchmark_data_smithi134_462957_object228:head# 2024-04-25T09:33:56.863 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fb20bb8f:::benchmark_data_smithi134_462957_object190:head# 2024-04-25T09:33:56.864 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fb6d87d7:::benchmark_data_smithi134_468469_object105:head# 2024-04-25T09:33:56.864 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fc414211:::benchmark_data_smithi134_462957_object79:head# 2024-04-25T09:33:56.865 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fc7f33f8:::benchmark_data_smithi134_462957_object37:head# 2024-04-25T09:33:56.865 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fe21a7b2:::benchmark_data_smithi134_468469_object68:head# 2024-04-25T09:33:56.866 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fe933281:::benchmark_data_smithi134_468469_object6:head# 2024-04-25T09:33:56.866 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fee7f4ff:::benchmark_data_smithi134_462957_object204:head# 2024-04-25T09:33:56.867 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ff13f1d8:::benchmark_data_smithi134_462957_object49:head# 2024-04-25T09:33:56.867 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ff758221:::benchmark_data_smithi134_468469_object56:head# 2024-04-25T09:33:56.868 INFO:tasks.workunit.client.0.smithi134.stderr:Export successful 2024-04-25T09:33:56.869 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T09:33:56.869 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T09:33:56.869 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:33:57.368 INFO:tasks.workunit.client.0.smithi134.stderr:Remove successful 2024-04-25T09:33:57.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:143: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 2024-04-25T09:33:58.503 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-04-25T09:33:58.503 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.1 2024-04-25T09:33:58.504 INFO:tasks.workunit.client.0.smithi134.stderr:PG export's map 9 is older than OSD's oldest_map 61 2024-04-25T09:33:58.504 INFO:tasks.workunit.client.0.smithi134.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:33:59.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:144: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:33:59.538 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-04-25T09:33:59.539 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.0 2024-04-25T09:33:59.539 INFO:tasks.workunit.client.0.smithi134.stderr:PG export's map 9 is older than OSD's oldest_map 61 2024-04-25T09:33:59.539 INFO:tasks.workunit.client.0.smithi134.stderr: pass --force to proceed anyway (with incomplete PastIntervals) 2024-04-25T09:34:00.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:147: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.1 --file td/pg-split-merge/1.1 --force 2024-04-25T09:34:00.580 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-04-25T09:34:00.581 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.1 2024-04-25T09:34:00.581 INFO:tasks.workunit.client.0.smithi134.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:34:00.581 INFO:tasks.workunit.client.0.smithi134.stderr:PG export's map 9 is older than OSD's oldest_map 61 2024-04-25T09:34:00.582 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:80f5f423:::benchmark_data_smithi134_462957_object61:head# 2024-04-25T09:34:00.582 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.610 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8136ddfc:::benchmark_data_smithi134_462957_object181:head# 2024-04-25T09:34:00.610 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.627 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:822e7d0d:::benchmark_data_smithi134_462957_object36:head# 2024-04-25T09:34:00.627 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.644 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:82344c27:::benchmark_data_smithi134_462957_object239:head# 2024-04-25T09:34:00.644 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.660 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:82ab8645:::benchmark_data_smithi134_462957_object312:head# 2024-04-25T09:34:00.660 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.677 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:83241f30:::benchmark_data_smithi134_462957_object186:head# 2024-04-25T09:34:00.677 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.694 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:833e47e2:::benchmark_data_smithi134_462957_object201:head# 2024-04-25T09:34:00.694 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.711 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:83909346:::benchmark_data_smithi134_462957_object165:head# 2024-04-25T09:34:00.711 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.727 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8486115e:::benchmark_data_smithi134_462957_object134:head# 2024-04-25T09:34:00.727 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.744 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:85d49e01:::benchmark_data_smithi134_462957_object114:head# 2024-04-25T09:34:00.744 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.761 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:86e69dbb:::benchmark_data_smithi134_462957_object334:head# 2024-04-25T09:34:00.761 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.777 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:87789c1f:::benchmark_data_smithi134_462957_object299:head# 2024-04-25T09:34:00.777 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.794 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:87e9176e:::benchmark_data_smithi134_462957_object176:head# 2024-04-25T09:34:00.794 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.811 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:88b81233:::benchmark_data_smithi134_462957_object197:head# 2024-04-25T09:34:00.811 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.827 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8a55344d:::benchmark_data_smithi134_462957_object163:head# 2024-04-25T09:34:00.827 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.844 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8b35579f:::benchmark_data_smithi134_462957_object101:head# 2024-04-25T09:34:00.844 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.861 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c42d79e:::benchmark_data_smithi134_462957_object285:head# 2024-04-25T09:34:00.861 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.877 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c5cb74b:::benchmark_data_smithi134_462957_object30:head# 2024-04-25T09:34:00.877 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.894 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c777440:::benchmark_data_smithi134_462957_object69:head# 2024-04-25T09:34:00.894 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.911 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8d18392e:::benchmark_data_smithi134_462957_object42:head# 2024-04-25T09:34:00.911 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.927 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8d4a51df:::benchmark_data_smithi134_462957_object202:head# 2024-04-25T09:34:00.927 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.944 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8f9fc238:::benchmark_data_smithi134_462957_object222:head# 2024-04-25T09:34:00.944 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.961 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8fcd40aa:::benchmark_data_smithi134_462957_object13:head# 2024-04-25T09:34:00.961 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.977 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8ffd53b9:::benchmark_data_smithi134_462957_object152:head# 2024-04-25T09:34:00.978 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:00.994 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:912ceabf:::benchmark_data_smithi134_462957_object291:head# 2024-04-25T09:34:00.994 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.011 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:913e15cc:::benchmark_data_smithi134_462957_object283:head# 2024-04-25T09:34:01.011 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.028 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:92bf16d1:::benchmark_data_smithi134_462957_object195:head# 2024-04-25T09:34:01.028 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.044 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:92ed1f0f:::benchmark_data_smithi134_462957_object65:head# 2024-04-25T09:34:01.044 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.069 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9417108d:::benchmark_data_smithi134_462957_object261:head# 2024-04-25T09:34:01.069 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.094 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:954cc32c:::benchmark_data_smithi134_462957_object139:head# 2024-04-25T09:34:01.094 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.111 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9632b147:::benchmark_data_smithi134_462957_object154:head# 2024-04-25T09:34:01.111 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.128 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9726b3f5:::benchmark_data_smithi134_462957_object135:head# 2024-04-25T09:34:01.128 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.144 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:97aa80c9:::benchmark_data_smithi134_462957_object210:head# 2024-04-25T09:34:01.144 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.161 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9870b61d:::benchmark_data_smithi134_462957_object50:head# 2024-04-25T09:34:01.161 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.178 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9a3c902a:::benchmark_data_smithi134_462957_object66:head# 2024-04-25T09:34:01.178 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.194 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9a63a41b:::benchmark_data_smithi134_462957_object59:head# 2024-04-25T09:34:01.195 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.211 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9af4c21c:::benchmark_data_smithi134_462957_object99:head# 2024-04-25T09:34:01.211 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.228 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9c02c888:::benchmark_data_smithi134_462957_object100:head# 2024-04-25T09:34:01.228 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.245 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9d740ac7:::benchmark_data_smithi134_462957_object156:head# 2024-04-25T09:34:01.245 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.261 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e32be9a:::benchmark_data_smithi134_462957_object54:head# 2024-04-25T09:34:01.261 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.278 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e6056cd:::benchmark_data_smithi134_462957_object164:head# 2024-04-25T09:34:01.278 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.294 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9ebf06db:::benchmark_data_smithi134_462957_object284:head# 2024-04-25T09:34:01.295 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.311 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a045e8d8:::benchmark_data_smithi134_462957_object234:head# 2024-04-25T09:34:01.311 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.328 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a08b0a4d:::benchmark_data_smithi134_462957_object241:head# 2024-04-25T09:34:01.328 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.344 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a0c022f8:::benchmark_data_smithi134_462957_object226:head# 2024-04-25T09:34:01.345 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.386 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a1658627:::benchmark_data_smithi134_462957_object327:head# 2024-04-25T09:34:01.386 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.403 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a1bb4fbf:::benchmark_data_smithi134_462957_object198:head# 2024-04-25T09:34:01.403 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.420 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a2b4a5bd:::benchmark_data_smithi134_462957_object27:head# 2024-04-25T09:34:01.420 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.436 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a5ca7c00:::benchmark_data_smithi134_462957_object72:head# 2024-04-25T09:34:01.436 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.453 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a5f62c63:::benchmark_data_smithi134_462957_object316:head# 2024-04-25T09:34:01.453 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.470 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a63f834f:::benchmark_data_smithi134_462957_object157:head# 2024-04-25T09:34:01.470 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.486 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a715009b:::benchmark_data_smithi134_462957_object169:head# 2024-04-25T09:34:01.487 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.503 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a85dbb56:::benchmark_data_smithi134_462957_object307:head# 2024-04-25T09:34:01.503 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.520 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a8d1cd5f:::benchmark_data_smithi134_462957_object107:head# 2024-04-25T09:34:01.520 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.536 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a9621e84:::benchmark_data_smithi134_462957_object303:head# 2024-04-25T09:34:01.536 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.553 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a9a511c7:::benchmark_data_smithi134_462957_object321:head# 2024-04-25T09:34:01.553 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.570 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ab6edda1:::benchmark_data_smithi134_462957_object212:head# 2024-04-25T09:34:01.570 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.586 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:aba338b0:::benchmark_data_smithi134_462957_object184:head# 2024-04-25T09:34:01.586 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.603 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:abdd83aa:::benchmark_data_smithi134_462957_object85:head# 2024-04-25T09:34:01.603 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.620 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ac85576b:::benchmark_data_smithi134_462957_object282:head# 2024-04-25T09:34:01.620 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.671 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ae25ce97:::benchmark_data_smithi134_462957_object75:head# 2024-04-25T09:34:01.672 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.712 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b1f8b3b5:::benchmark_data_smithi134_462957_object160:head# 2024-04-25T09:34:01.712 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.728 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b249419b:::benchmark_data_smithi134_462957_object275:head# 2024-04-25T09:34:01.728 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.745 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b289940b:::benchmark_data_smithi134_462957_object26:head# 2024-04-25T09:34:01.745 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.761 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b29d1223:::benchmark_data_smithi134_462957_object167:head# 2024-04-25T09:34:01.761 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.787 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b4ae5688:::benchmark_data_smithi134_462957_object182:head# 2024-04-25T09:34:01.787 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.803 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b5565581:::benchmark_data_smithi134_462957_object22:head# 2024-04-25T09:34:01.803 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.820 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b56627a7:::benchmark_data_smithi134_462957_object108:head# 2024-04-25T09:34:01.820 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.836 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b6ff86cd:::benchmark_data_smithi134_462957_object166:head# 2024-04-25T09:34:01.837 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.853 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b7edf419:::benchmark_data_smithi134_462957_object84:head# 2024-04-25T09:34:01.853 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.870 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:34:01.870 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.887 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b8f154ff:::benchmark_data_smithi134_462957_object206:head# 2024-04-25T09:34:01.887 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.903 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b93b7738:::benchmark_data_smithi134_462957_object92:head# 2024-04-25T09:34:01.903 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.920 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b997874c:::benchmark_data_smithi134_462957_object320:head# 2024-04-25T09:34:01.920 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.937 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ba34c80e:::benchmark_data_smithi134_462957_object29:head# 2024-04-25T09:34:01.937 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.953 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bb51ab09:::benchmark_data_smithi134_462957_object246:head# 2024-04-25T09:34:01.953 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.970 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bb9ed292:::benchmark_data_smithi134_462957_object44:head# 2024-04-25T09:34:01.970 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:01.987 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bc0fe2eb:::benchmark_data_smithi134_462957_object105:head# 2024-04-25T09:34:01.987 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.003 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bd33c13c:::benchmark_data_smithi134_462957_object121:head# 2024-04-25T09:34:02.003 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.020 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bd52ec16:::benchmark_data_smithi134_462957_object68:head# 2024-04-25T09:34:02.020 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.037 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:be19e778:::benchmark_data_smithi134_462957_object238:head# 2024-04-25T09:34:02.037 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.077 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bf64a100:::benchmark_data_smithi134_462957_object83:head# 2024-04-25T09:34:02.077 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.127 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c03f464f:::benchmark_data_smithi134_462957_object18:head# 2024-04-25T09:34:02.127 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.185 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c0f283ce:::benchmark_data_smithi134_462957_object124:head# 2024-04-25T09:34:02.185 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.220 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c16c7b78:::benchmark_data_smithi134_462957_object2:head# 2024-04-25T09:34:02.220 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.237 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c18f6438:::benchmark_data_smithi134_462957_object220:head# 2024-04-25T09:34:02.237 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.253 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c244846a:::benchmark_data_smithi134_462957_object329:head# 2024-04-25T09:34:02.254 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.270 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c274bcc9:::benchmark_data_smithi134_462957_object332:head# 2024-04-25T09:34:02.270 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.287 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c2e9f6b4:::benchmark_data_smithi134_462957_object330:head# 2024-04-25T09:34:02.287 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.304 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c35cfddf:::benchmark_data_smithi134_462957_object200:head# 2024-04-25T09:34:02.304 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.320 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c47b78d0:::benchmark_data_smithi134_462957_object117:head# 2024-04-25T09:34:02.320 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.337 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c70a7496:::benchmark_data_smithi134_462957_object60:head# 2024-04-25T09:34:02.337 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.354 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c73308dd:::benchmark_data_smithi134_462957_object258:head# 2024-04-25T09:34:02.354 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.370 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c74b145d:::benchmark_data_smithi134_462957_object146:head# 2024-04-25T09:34:02.371 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.387 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c7d37ef1:::benchmark_data_smithi134_462957_object125:head# 2024-04-25T09:34:02.387 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.404 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c84d6c63:::benchmark_data_smithi134_462957_object221:head# 2024-04-25T09:34:02.404 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.420 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c8d637f7:::benchmark_data_smithi134_462957_object306:head# 2024-04-25T09:34:02.421 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.437 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c9e347b1:::benchmark_data_smithi134_462957_object242:head# 2024-04-25T09:34:02.437 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.454 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ca46b863:::benchmark_data_smithi134_462957_object57:head# 2024-04-25T09:34:02.454 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.471 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ca795a74:::benchmark_data_smithi134_462957_object81:head# 2024-04-25T09:34:02.471 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.487 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cc49d25d:::benchmark_data_smithi134_462957_object144:head# 2024-04-25T09:34:02.487 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.504 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ccd8cbfe:::benchmark_data_smithi134_462957_object5:head# 2024-04-25T09:34:02.504 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.520 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cce05002:::benchmark_data_smithi134_462957_object130:head# 2024-04-25T09:34:02.521 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.537 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cd79c5f8:::benchmark_data_smithi134_462957_object76:head# 2024-04-25T09:34:02.537 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.554 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cdb281d5:::benchmark_data_smithi134_462957_object46:head# 2024-04-25T09:34:02.554 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.571 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cdc69e54:::benchmark_data_smithi134_462957_object252:head# 2024-04-25T09:34:02.571 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.587 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf2a4062:::benchmark_data_smithi134_462957_object199:head# 2024-04-25T09:34:02.587 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.604 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf35812c:::benchmark_data_smithi134_462957_object243:head# 2024-04-25T09:34:02.604 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.621 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf693ea1:::benchmark_data_smithi134_462957_object147:head# 2024-04-25T09:34:02.621 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.637 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d0dad50c:::benchmark_data_smithi134_462957_object97:head# 2024-04-25T09:34:02.637 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.654 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d1761cb1:::benchmark_data_smithi134_462957_object251:head# 2024-04-25T09:34:02.654 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.671 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d1b7fc50:::benchmark_data_smithi134_462957_object211:head# 2024-04-25T09:34:02.671 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.687 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d2bd3105:::benchmark_data_smithi134_462957_object277:head# 2024-04-25T09:34:02.687 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.704 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d3201ed2:::benchmark_data_smithi134_462957_object45:head# 2024-04-25T09:34:02.704 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.721 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d423debd:::benchmark_data_smithi134_462957_object115:head# 2024-04-25T09:34:02.721 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.737 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d4c4f287:::benchmark_data_smithi134_462957_object21:head# 2024-04-25T09:34:02.737 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.755 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d50a78a1:::benchmark_data_smithi134_462957_object25:head# 2024-04-25T09:34:02.755 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.771 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d5af3bd1:::benchmark_data_smithi134_462957_object180:head# 2024-04-25T09:34:02.771 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.787 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d5ef9c20:::benchmark_data_smithi134_462957_object70:head# 2024-04-25T09:34:02.788 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.804 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d65dc781:::benchmark_data_smithi134_462957_object323:head# 2024-04-25T09:34:02.804 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.821 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6872c2f:::benchmark_data_smithi134_462957_object192:head# 2024-04-25T09:34:02.821 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.838 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6cbb661:::benchmark_data_smithi134_462957_object279:head# 2024-04-25T09:34:02.838 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.854 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6d8a7de:::benchmark_data_smithi134_462957_object175:head# 2024-04-25T09:34:02.854 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.871 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d9df41a0:::benchmark_data_smithi134_462957_object288:head# 2024-04-25T09:34:02.871 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.888 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dba2aa95:::benchmark_data_smithi134_462957_object31:head# 2024-04-25T09:34:02.888 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.904 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dc540d1b:::benchmark_data_smithi134_462957_object189:head# 2024-04-25T09:34:02.904 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.921 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dc6e92dd:::benchmark_data_smithi134_462957_object159:head# 2024-04-25T09:34:02.921 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.938 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dcca6922:::benchmark_data_smithi134_462957_object311:head# 2024-04-25T09:34:02.938 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:02.954 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd1204a0:::benchmark_data_smithi134_462957_object71:head# 2024-04-25T09:34:02.954 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.005 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd3e6d82:::benchmark_data_smithi134_462957_object262:head# 2024-04-25T09:34:03.005 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.021 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dfc1c8cf:::benchmark_data_smithi134_462957_object229:head# 2024-04-25T09:34:03.022 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.038 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e06b8840:::benchmark_data_smithi134_462957_object91:head# 2024-04-25T09:34:03.038 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.054 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e06cab0a:::benchmark_data_smithi134_462957_object23:head# 2024-04-25T09:34:03.054 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.071 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e28f6f75:::benchmark_data_smithi134_462957_object300:head# 2024-04-25T09:34:03.071 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.088 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e2be0804:::benchmark_data_smithi134_462957_object235:head# 2024-04-25T09:34:03.088 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.105 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e2c8c0bf:::benchmark_data_smithi134_462957_object40:head# 2024-04-25T09:34:03.105 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.122 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e3a40c39:::benchmark_data_smithi134_462957_object63:head# 2024-04-25T09:34:03.122 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.141 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e5228945:::benchmark_data_smithi134_462957_object293:head# 2024-04-25T09:34:03.141 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.154 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e614399c:::benchmark_data_smithi134_462957_object110:head# 2024-04-25T09:34:03.155 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.172 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e65f056d:::benchmark_data_smithi134_462957_object196:head# 2024-04-25T09:34:03.172 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.188 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e6a6c722:::benchmark_data_smithi134_462957_object264:head# 2024-04-25T09:34:03.188 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.205 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e74bd302:::benchmark_data_smithi134_462957_object131:head# 2024-04-25T09:34:03.205 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.221 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e7aa26a1:::benchmark_data_smithi134_462957_object129:head# 2024-04-25T09:34:03.221 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.238 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e8378195:::benchmark_data_smithi134_462957_object256:head# 2024-04-25T09:34:03.238 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.255 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e8ce525c:::benchmark_data_smithi134_462957_object155:head# 2024-04-25T09:34:03.255 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.271 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e8f5abd5:::benchmark_data_smithi134_462957_object56:head# 2024-04-25T09:34:03.272 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.288 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e985cf7e:::benchmark_data_smithi134_462957_object287:head# 2024-04-25T09:34:03.288 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.305 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e9f08f42:::benchmark_data_smithi134_462957_object187:head# 2024-04-25T09:34:03.305 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.321 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ea484b6f:::benchmark_data_smithi134_462957_object82:head# 2024-04-25T09:34:03.322 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.338 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ee73ec6d:::benchmark_data_smithi134_462957_object33:head# 2024-04-25T09:34:03.338 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.355 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f01022da:::benchmark_data_smithi134_462957_object324:head# 2024-04-25T09:34:03.355 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.371 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0800d50:::benchmark_data_smithi134_462957_object217:head# 2024-04-25T09:34:03.372 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.388 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0f00dfc:::benchmark_data_smithi134_462957_object185:head# 2024-04-25T09:34:03.388 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.405 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0f2cea2:::benchmark_data_smithi134_462957_object317:head# 2024-04-25T09:34:03.405 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.421 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f131ea49:::benchmark_data_smithi134_462957_object15:head# 2024-04-25T09:34:03.422 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.438 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f3c72ea5:::benchmark_data_smithi134_462957_object173:head# 2024-04-25T09:34:03.439 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.455 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f581c1b4:::benchmark_data_smithi134_462957_object218:head# 2024-04-25T09:34:03.455 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.471 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f63f8b4d:::benchmark_data_smithi134_462957_object247:head# 2024-04-25T09:34:03.472 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.488 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f70159a7:::benchmark_data_smithi134_462957_object208:head# 2024-04-25T09:34:03.488 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.505 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f741ec8b:::benchmark_data_smithi134_462957_object325:head# 2024-04-25T09:34:03.505 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.521 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f7f2b0bd:::benchmark_data_smithi134_462957_object302:head# 2024-04-25T09:34:03.522 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.538 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f8d2618b:::benchmark_data_smithi134_462957_object255:head# 2024-04-25T09:34:03.539 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.555 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f8f97fba:::benchmark_data_smithi134_462957_object178:head# 2024-04-25T09:34:03.555 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.572 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fa5b5d21:::benchmark_data_smithi134_462957_object274:head# 2024-04-25T09:34:03.572 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.588 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:faa372e4:::benchmark_data_smithi134_462957_object228:head# 2024-04-25T09:34:03.588 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.605 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fb20bb8f:::benchmark_data_smithi134_462957_object190:head# 2024-04-25T09:34:03.605 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.622 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fc414211:::benchmark_data_smithi134_462957_object79:head# 2024-04-25T09:34:03.622 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.680 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fc7f33f8:::benchmark_data_smithi134_462957_object37:head# 2024-04-25T09:34:03.680 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.722 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fee7f4ff:::benchmark_data_smithi134_462957_object204:head# 2024-04-25T09:34:03.722 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.738 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ff13f1d8:::benchmark_data_smithi134_462957_object49:head# 2024-04-25T09:34:03.738 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:03.755 INFO:tasks.workunit.client.0.smithi134.stdout:write_pg epoch 9 info 1.1( v 9'170 (0'0,9'170] local-lis/les=7/8 n=170 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:34:03.772 INFO:tasks.workunit.client.0.smithi134.stdout:Import successful 2024-04-25T09:34:04.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:148: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:34:04.966 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-04-25T09:34:04.966 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.0 2024-04-25T09:34:04.967 INFO:tasks.workunit.client.0.smithi134.stdout:pool 1 pg_num 2 -> 1 2024-04-25T09:34:04.967 INFO:tasks.workunit.client.0.smithi134.stderr:PG export's map 9 is older than OSD's oldest_map 61 2024-04-25T09:34:04.968 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:014fbc01:::benchmark_data_smithi134_462957_object116:head# 2024-04-25T09:34:04.968 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.006 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:01ae4a14:::benchmark_data_smithi134_462957_object335:head# 2024-04-25T09:34:05.006 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.031 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:022eaaef:::benchmark_data_smithi134_462957_object132:head# 2024-04-25T09:34:05.031 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.047 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:04df5908:::benchmark_data_smithi134_462957_object10:head# 2024-04-25T09:34:05.048 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.072 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:05e918e6:::benchmark_data_smithi134_462957_object87:head# 2024-04-25T09:34:05.073 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.097 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:08057fd0:::benchmark_data_smithi134_462957_object95:head# 2024-04-25T09:34:05.098 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.122 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:08313bb1:::benchmark_data_smithi134_462957_object240:head# 2024-04-25T09:34:05.123 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.148 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0958de82:::benchmark_data_smithi134_462957_object305:head# 2024-04-25T09:34:05.148 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.173 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:09655936:::benchmark_data_smithi134_462957_object278:head# 2024-04-25T09:34:05.173 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.239 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0a7e421b:::benchmark_data_smithi134_462957_object39:head# 2024-04-25T09:34:05.240 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.264 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0be5f62e:::benchmark_data_smithi134_462957_object126:head# 2024-04-25T09:34:05.265 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.289 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0ce6c2ff:::benchmark_data_smithi134_462957_object48:head# 2024-04-25T09:34:05.290 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.315 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0d00a542:::benchmark_data_smithi134_462957_object112:head# 2024-04-25T09:34:05.315 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.339 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0d256506:::benchmark_data_smithi134_462957_object177:head# 2024-04-25T09:34:05.340 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.364 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0e18ad9b:::benchmark_data_smithi134_462957_object52:head# 2024-04-25T09:34:05.365 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.390 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0f50edf7:::benchmark_data_smithi134_462957_object304:head# 2024-04-25T09:34:05.390 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.415 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0f680139:::benchmark_data_smithi134_462957_object310:head# 2024-04-25T09:34:05.415 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.465 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0fc29821:::benchmark_data_smithi134_462957_object276:head# 2024-04-25T09:34:05.465 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.481 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:109d776b:::benchmark_data_smithi134_462957_object331:head# 2024-04-25T09:34:05.481 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.498 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1131e4a2:::benchmark_data_smithi134_462957_object266:head# 2024-04-25T09:34:05.498 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.515 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:11a2460d:::benchmark_data_smithi134_462957_object267:head# 2024-04-25T09:34:05.515 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.531 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1255ae71:::benchmark_data_smithi134_462957_object123:head# 2024-04-25T09:34:05.531 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.548 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:134c0c2c:::benchmark_data_smithi134_462957_object140:head# 2024-04-25T09:34:05.548 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.565 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:136d3c79:::benchmark_data_smithi134_462957_object216:head# 2024-04-25T09:34:05.565 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.581 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:14c28473:::benchmark_data_smithi134_462957_object143:head# 2024-04-25T09:34:05.581 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.598 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:14d46dea:::benchmark_data_smithi134_462957_object322:head# 2024-04-25T09:34:05.598 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.615 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:151932bc:::benchmark_data_smithi134_462957_object127:head# 2024-04-25T09:34:05.615 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.645 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:15788a9d:::benchmark_data_smithi134_462957_object215:head# 2024-04-25T09:34:05.645 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.703 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1607bd5a:::benchmark_data_smithi134_462957_object24:head# 2024-04-25T09:34:05.703 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.723 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:16357f0a:::benchmark_data_smithi134_462957_object16:head# 2024-04-25T09:34:05.723 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.740 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:16baa595:::benchmark_data_smithi134_462957_object309:head# 2024-04-25T09:34:05.740 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.756 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1965afdf:::benchmark_data_smithi134_462957_object89:head# 2024-04-25T09:34:05.756 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.773 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1affb4ca:::benchmark_data_smithi134_462957_object62:head# 2024-04-25T09:34:05.773 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.790 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1b230e12:::benchmark_data_smithi134_462957_object118:head# 2024-04-25T09:34:05.790 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.806 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1bd04fc7:::benchmark_data_smithi134_462957_object188:head# 2024-04-25T09:34:05.807 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.823 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1d14325e:::benchmark_data_smithi134_462957_object41:head# 2024-04-25T09:34:05.823 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.842 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1d8f10c1:::benchmark_data_smithi134_462957_object86:head# 2024-04-25T09:34:05.842 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.859 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1da2d015:::benchmark_data_smithi134_462957_object64:head# 2024-04-25T09:34:05.859 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.875 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1dad9b9c:::benchmark_data_smithi134_462957_object257:head# 2024-04-25T09:34:05.876 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.892 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1fb02fa0:::benchmark_data_smithi134_462957_object193:head# 2024-04-25T09:34:05.892 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.909 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:203c2195:::benchmark_data_smithi134_462957_object254:head# 2024-04-25T09:34:05.909 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.926 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:22302b8d:::benchmark_data_smithi134_462957_object80:head# 2024-04-25T09:34:05.926 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.942 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:228f6133:::benchmark_data_smithi134_462957_object74:head# 2024-04-25T09:34:05.942 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.959 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:23e9d7b5:::benchmark_data_smithi134_462957_object245:head# 2024-04-25T09:34:05.959 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.976 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2426c803:::benchmark_data_smithi134_462957_object290:head# 2024-04-25T09:34:05.976 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:05.992 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:244ef85c:::benchmark_data_smithi134_462957_object170:head# 2024-04-25T09:34:05.993 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.009 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2513bbe2:::benchmark_data_smithi134_462957_object207:head# 2024-04-25T09:34:06.009 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.026 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2568f8b5:::benchmark_data_smithi134_462957_object8:head# 2024-04-25T09:34:06.026 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.042 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:260f1533:::benchmark_data_smithi134_462957_object191:head# 2024-04-25T09:34:06.042 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.093 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:27170e69:::benchmark_data_smithi134_462957_object34:head# 2024-04-25T09:34:06.093 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.126 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:275fd7c3:::benchmark_data_smithi134_462957_object225:head# 2024-04-25T09:34:06.126 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.184 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:28ef7a99:::benchmark_data_smithi134_462957_object162:head# 2024-04-25T09:34:06.184 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.209 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2b3177c6:::benchmark_data_smithi134_462957_object298:head# 2024-04-25T09:34:06.209 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.226 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2b9356c2:::benchmark_data_smithi134_462957_object270:head# 2024-04-25T09:34:06.226 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.242 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2cb634ef:::benchmark_data_smithi134_462957_object297:head# 2024-04-25T09:34:06.242 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.259 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2ccb43ac:::benchmark_data_smithi134_462957_object28:head# 2024-04-25T09:34:06.259 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.276 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2cf40efb:::benchmark_data_smithi134_462957_object289:head# 2024-04-25T09:34:06.276 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.292 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2d9f151f:::benchmark_data_smithi134_462957_object73:head# 2024-04-25T09:34:06.292 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.309 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2dfb304e:::benchmark_data_smithi134_462957_object138:head# 2024-04-25T09:34:06.309 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.326 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2e76756e:::benchmark_data_smithi134_462957_object174:head# 2024-04-25T09:34:06.326 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.343 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2f2a62da:::benchmark_data_smithi134_462957_object230:head# 2024-04-25T09:34:06.343 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.359 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2f954ef6:::benchmark_data_smithi134_462957_object120:head# 2024-04-25T09:34:06.359 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.376 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:302de82f:::benchmark_data_smithi134_462957_object268:head# 2024-04-25T09:34:06.376 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.393 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:314aacf8:::benchmark_data_smithi134_462957_object98:head# 2024-04-25T09:34:06.393 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.409 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:328e347c:::benchmark_data_smithi134_462957_object203:head# 2024-04-25T09:34:06.409 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.435 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:33056cad:::benchmark_data_smithi134_462957_object248:head# 2024-04-25T09:34:06.435 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.451 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3342be62:::benchmark_data_smithi134_462957_object158:head# 2024-04-25T09:34:06.452 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.468 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:33c19275:::benchmark_data_smithi134_462957_object43:head# 2024-04-25T09:34:06.468 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.484 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:34187219:::benchmark_data_smithi134_462957_object294:head# 2024-04-25T09:34:06.485 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.501 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:344b976e:::benchmark_data_smithi134_462957_object231:head# 2024-04-25T09:34:06.501 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.518 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:358811ac:::benchmark_data_smithi134_462957_object142:head# 2024-04-25T09:34:06.518 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.535 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:361e89fa:::benchmark_data_smithi134_462957_object308:head# 2024-04-25T09:34:06.535 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.551 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:369c019f:::benchmark_data_smithi134_462957_object313:head# 2024-04-25T09:34:06.551 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.568 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:39249c23:::benchmark_data_smithi134_462957_object161:head# 2024-04-25T09:34:06.568 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.584 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3a8ee0cf:::benchmark_data_smithi134_462957_object151:head# 2024-04-25T09:34:06.584 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.601 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3b9ee75a:::benchmark_data_smithi134_462957_object326:head# 2024-04-25T09:34:06.601 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.618 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3e5f4585:::benchmark_data_smithi134_462957_object47:head# 2024-04-25T09:34:06.618 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.634 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3ed5c4c3:::benchmark_data_smithi134_462957_object205:head# 2024-04-25T09:34:06.635 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.651 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3fc6ca95:::benchmark_data_smithi134_462957_object12:head# 2024-04-25T09:34:06.651 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.668 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:40470fd1:::benchmark_data_smithi134_462957_object17:head# 2024-04-25T09:34:06.668 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.684 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:417961ce:::benchmark_data_smithi134_462957_object319:head# 2024-04-25T09:34:06.685 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.701 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:418ec0ee:::benchmark_data_smithi134_462957_object149:head# 2024-04-25T09:34:06.701 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.718 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:422d92f6:::benchmark_data_smithi134_462957_object272:head# 2024-04-25T09:34:06.718 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.735 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:435e2086:::benchmark_data_smithi134_462957_object171:head# 2024-04-25T09:34:06.735 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.751 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:444a49fd:::benchmark_data_smithi134_462957_object328:head# 2024-04-25T09:34:06.751 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.768 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:44a8dd41:::benchmark_data_smithi134_462957_object3:head# 2024-04-25T09:34:06.768 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.785 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:467046b5:::benchmark_data_smithi134_462957_object265:head# 2024-04-25T09:34:06.785 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.801 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4672b82d:::benchmark_data_smithi134_462957_object295:head# 2024-04-25T09:34:06.801 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.818 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47196e71:::benchmark_data_smithi134_462957_object269:head# 2024-04-25T09:34:06.818 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.835 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:478e238f:::benchmark_data_smithi134_462957_object213:head# 2024-04-25T09:34:06.835 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.851 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47e25b7f:::benchmark_data_smithi134_462957_object249:head# 2024-04-25T09:34:06.851 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.869 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47f5dd1f:::benchmark_data_smithi134_462957_object122:head# 2024-04-25T09:34:06.869 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.885 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:49e180d1:::benchmark_data_smithi134_462957_object88:head# 2024-04-25T09:34:06.885 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.902 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4a676de3:::benchmark_data_smithi134_462957_object263:head# 2024-04-25T09:34:06.902 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.918 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4b79b182:::benchmark_data_smithi134_462957_object137:head# 2024-04-25T09:34:06.919 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.935 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4cb6f1b7:::benchmark_data_smithi134_462957_object78:head# 2024-04-25T09:34:06.935 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.952 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4e62c3c1:::benchmark_data_smithi134_462957_object53:head# 2024-04-25T09:34:06.952 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.968 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4e981d63:::benchmark_data_smithi134_462957_object227:head# 2024-04-25T09:34:06.968 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:06.985 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4fda476d:::benchmark_data_smithi134_462957_object194:head# 2024-04-25T09:34:06.985 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.001 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:504c6844:::benchmark_data_smithi134_462957_object94:head# 2024-04-25T09:34:07.002 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.018 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:504ee13c:::benchmark_data_smithi134_462957_object93:head# 2024-04-25T09:34:07.018 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.035 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:511d8705:::benchmark_data_smithi134_462957_object271:head# 2024-04-25T09:34:07.035 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.052 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:51b089df:::benchmark_data_smithi134_462957_object273:head# 2024-04-25T09:34:07.052 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.068 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5266c200:::benchmark_data_smithi134_462957_object32:head# 2024-04-25T09:34:07.069 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.085 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:542d36c1:::benchmark_data_smithi134_462957_object4:head# 2024-04-25T09:34:07.086 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.102 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:544671af:::benchmark_data_smithi134_462957_object172:head# 2024-04-25T09:34:07.102 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.118 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:549eba21:::benchmark_data_smithi134_462957_object133:head# 2024-04-25T09:34:07.118 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.135 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:556d8a79:::benchmark_data_smithi134_462957_object214:head# 2024-04-25T09:34:07.135 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.152 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:556e8b3c:::benchmark_data_smithi134_462957_object96:head# 2024-04-25T09:34:07.152 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.168 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:55950821:::benchmark_data_smithi134_462957_object141:head# 2024-04-25T09:34:07.168 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.177 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:575ccb6b:::benchmark_data_smithi134_462957_object280:head# 2024-04-25T09:34:07.177 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.185 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:57a020db:::benchmark_data_smithi134_462957_object286:head# 2024-04-25T09:34:07.185 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.194 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:58587487:::benchmark_data_smithi134_462957_object51:head# 2024-04-25T09:34:07.194 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.210 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5c373486:::benchmark_data_smithi134_462957_object14:head# 2024-04-25T09:34:07.210 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.228 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5dada283:::benchmark_data_smithi134_462957_object77:head# 2024-04-25T09:34:07.228 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.244 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5e48ae44:::benchmark_data_smithi134_462957_object333:head# 2024-04-25T09:34:07.244 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.260 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:60d506d0:::benchmark_data_smithi134_462957_object111:head# 2024-04-25T09:34:07.261 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.277 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:61342236:::benchmark_data_smithi134_462957_object301:head# 2024-04-25T09:34:07.277 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.294 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:61d1b2a4:::benchmark_data_smithi134_462957_object179:head# 2024-04-25T09:34:07.294 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.310 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:621d43c5:::benchmark_data_smithi134_462957_object102:head# 2024-04-25T09:34:07.311 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.327 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:62545d41:::benchmark_data_smithi134_462957_object318:head# 2024-04-25T09:34:07.327 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.344 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:632b2ff2:::benchmark_data_smithi134_462957_object113:head# 2024-04-25T09:34:07.344 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.360 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:63effe0b:::benchmark_data_smithi134_462957_object1:head# 2024-04-25T09:34:07.360 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.377 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:63f38e63:::benchmark_data_smithi134_462957_object314:head# 2024-04-25T09:34:07.377 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.394 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:645caf70:::benchmark_data_smithi134_462957_object9:head# 2024-04-25T09:34:07.394 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.410 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:64cb656a:::benchmark_data_smithi134_462957_object38:head# 2024-04-25T09:34:07.411 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.427 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:651d1c11:::benchmark_data_smithi134_462957_object219:head# 2024-04-25T09:34:07.427 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.444 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:65698cf1:::benchmark_data_smithi134_462957_object315:head# 2024-04-25T09:34:07.444 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.452 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:660b1875:::benchmark_data_smithi134_462957_object153:head# 2024-04-25T09:34:07.452 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.511 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:66e4bb20:::benchmark_data_smithi134_462957_object259:head# 2024-04-25T09:34:07.511 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.527 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67406b53:::benchmark_data_smithi134_462957_object168:head# 2024-04-25T09:34:07.527 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.544 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6753f201:::benchmark_data_smithi134_462957_object19:head# 2024-04-25T09:34:07.544 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.561 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67de4446:::benchmark_data_smithi134_462957_object11:head# 2024-04-25T09:34:07.561 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.569 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68431e33:::benchmark_data_smithi134_462957_object6:head# 2024-04-25T09:34:07.569 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.577 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68da1af8:::benchmark_data_smithi134_462957_object224:head# 2024-04-25T09:34:07.577 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.586 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68e29e1b:::benchmark_data_smithi134_462957_object223:head# 2024-04-25T09:34:07.586 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.603 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68ff3868:::benchmark_data_smithi134_462957_object104:head# 2024-04-25T09:34:07.603 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.619 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:69444b58:::benchmark_data_smithi134_462957_object236:head# 2024-04-25T09:34:07.619 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.679 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6bef445e:::benchmark_data_smithi134_462957_object145:head# 2024-04-25T09:34:07.679 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.711 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6c6b0a68:::benchmark_data_smithi134_462957_object106:head# 2024-04-25T09:34:07.711 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.719 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6cb36ab9:::benchmark_data_smithi134_462957_object128:head# 2024-04-25T09:34:07.719 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.727 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6cb830fe:::benchmark_data_smithi134_462957_object281:head# 2024-04-25T09:34:07.727 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.736 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6d038ea7:::benchmark_data_smithi134_462957_object250:head# 2024-04-25T09:34:07.736 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.744 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6dfad7ef:::benchmark_data_smithi134_462957_object253:head# 2024-04-25T09:34:07.744 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.752 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6e8445ed:::benchmark_data_smithi134_462957_object119:head# 2024-04-25T09:34:07.752 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.761 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6ea927e3:::benchmark_data_smithi134_462957_object67:head# 2024-04-25T09:34:07.761 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.769 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6edbe582:::benchmark_data_smithi134_462957_object260:head# 2024-04-25T09:34:07.769 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.778 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6f922ade:::benchmark_data_smithi134_462957_object136:head# 2024-04-25T09:34:07.778 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.786 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6fe808b0:::benchmark_data_smithi134_462957_object7:head# 2024-04-25T09:34:07.786 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.794 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:71c24ec6:::benchmark_data_smithi134_462957_object35:head# 2024-04-25T09:34:07.794 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.802 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7280df13:::benchmark_data_smithi134_462957_object109:head# 2024-04-25T09:34:07.803 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.811 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:72b74ac7:::benchmark_data_smithi134_462957_object292:head# 2024-04-25T09:34:07.811 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.819 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:72d020e0:::benchmark_data_smithi134_462957_object20:head# 2024-04-25T09:34:07.819 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.827 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:73570709:::benchmark_data_smithi134_462957_object244:head# 2024-04-25T09:34:07.828 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.836 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:73fa9d47:::benchmark_data_smithi134_462957_object0:head# 2024-04-25T09:34:07.836 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.844 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:76387380:::benchmark_data_smithi134_462957_object103:head# 2024-04-25T09:34:07.844 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.853 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:77a61cee:::benchmark_data_smithi134_462957_object237:head# 2024-04-25T09:34:07.853 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.861 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7826dd0b:::benchmark_data_smithi134_462957_object233:head# 2024-04-25T09:34:07.861 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.869 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7958a491:::benchmark_data_smithi134_462957_object232:head# 2024-04-25T09:34:07.870 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.878 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7a5eed6c:::benchmark_data_smithi134_462957_object55:head# 2024-04-25T09:34:07.878 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.886 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7bed12e2:::benchmark_data_smithi134_462957_object90:head# 2024-04-25T09:34:07.886 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.894 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7c53f004:::benchmark_data_smithi134_462957_object183:head# 2024-04-25T09:34:07.895 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.903 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7cc751e9:::benchmark_data_smithi134_462957_object150:head# 2024-04-25T09:34:07.903 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.911 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7de067cb:::benchmark_data_smithi134_462957_object148:head# 2024-04-25T09:34:07.912 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.920 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7e10a619:::benchmark_data_smithi134_462957_object296:head# 2024-04-25T09:34:07.920 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.928 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7f2f5709:::benchmark_data_smithi134_462957_object58:head# 2024-04-25T09:34:07.928 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.936 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7f4d9c89:::benchmark_data_smithi134_462957_object209:head# 2024-04-25T09:34:07.936 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:07.945 INFO:tasks.workunit.client.0.smithi134.stdout:write_pg epoch 9 info 1.0( v 9'167 (0'0,9'167] local-lis/les=7/8 n=167 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:34:07.961 INFO:tasks.workunit.client.0.smithi134.stdout:Import successful 2024-04-25T09:34:08.497 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:151: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:34:09.196 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T09:34:09.197 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T09:34:09.197 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:34:09.270 INFO:tasks.workunit.client.0.smithi134.stdout:Remove successful 2024-04-25T09:34:09.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:152: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-04-25T09:34:10.432 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T09:34:10.432 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T09:34:10.432 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:34:10.504 INFO:tasks.workunit.client.0.smithi134.stdout:Remove successful 2024-04-25T09:34:10.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:154: TEST_import_after_merge_and_gap: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0.later --force 2024-04-25T09:34:11.690 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e228 pg_nums {1={19=1}} deleted_pools ) 2024-04-25T09:34:11.690 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.0 2024-04-25T09:34:11.690 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:00b7a8e8:::benchmark_data_smithi134_468469_object49:head# 2024-04-25T09:34:11.691 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.737 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:012cc06f:::benchmark_data_smithi134_468469_object113:head# 2024-04-25T09:34:11.737 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.754 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:014fbc01:::benchmark_data_smithi134_462957_object116:head# 2024-04-25T09:34:11.754 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.779 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:01ae4a14:::benchmark_data_smithi134_462957_object335:head# 2024-04-25T09:34:11.779 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.804 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:022eaaef:::benchmark_data_smithi134_462957_object132:head# 2024-04-25T09:34:11.804 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.821 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:031c8e30:::benchmark_data_smithi134_468469_object20:head# 2024-04-25T09:34:11.821 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.837 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:038827ab:::benchmark_data_smithi134_468469_object23:head# 2024-04-25T09:34:11.838 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.854 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:04df5908:::benchmark_data_smithi134_462957_object10:head# 2024-04-25T09:34:11.854 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.871 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:05e918e6:::benchmark_data_smithi134_462957_object87:head# 2024-04-25T09:34:11.871 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.887 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:08057fd0:::benchmark_data_smithi134_462957_object95:head# 2024-04-25T09:34:11.888 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.904 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:08313bb1:::benchmark_data_smithi134_462957_object240:head# 2024-04-25T09:34:11.904 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.921 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:093a7b43:::benchmark_data_smithi134_468469_object40:head# 2024-04-25T09:34:11.921 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.938 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0958de82:::benchmark_data_smithi134_462957_object305:head# 2024-04-25T09:34:11.938 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.954 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:09655936:::benchmark_data_smithi134_462957_object278:head# 2024-04-25T09:34:11.954 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.971 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0a7e421b:::benchmark_data_smithi134_462957_object39:head# 2024-04-25T09:34:11.971 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:11.988 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0b3dcfda:::benchmark_data_smithi134_468469_object83:head# 2024-04-25T09:34:11.988 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.004 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0be5f62e:::benchmark_data_smithi134_462957_object126:head# 2024-04-25T09:34:12.004 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.021 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0ce6c2ff:::benchmark_data_smithi134_462957_object48:head# 2024-04-25T09:34:12.021 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.038 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0d00a542:::benchmark_data_smithi134_462957_object112:head# 2024-04-25T09:34:12.038 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.081 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0d256506:::benchmark_data_smithi134_462957_object177:head# 2024-04-25T09:34:12.081 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.138 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0e18ad9b:::benchmark_data_smithi134_462957_object52:head# 2024-04-25T09:34:12.138 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.196 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0f50edf7:::benchmark_data_smithi134_462957_object304:head# 2024-04-25T09:34:12.196 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.221 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0f680139:::benchmark_data_smithi134_462957_object310:head# 2024-04-25T09:34:12.221 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.238 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0fc29821:::benchmark_data_smithi134_462957_object276:head# 2024-04-25T09:34:12.238 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.254 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:109d776b:::benchmark_data_smithi134_462957_object331:head# 2024-04-25T09:34:12.254 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.271 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1131e4a2:::benchmark_data_smithi134_462957_object266:head# 2024-04-25T09:34:12.271 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.288 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:11a2460d:::benchmark_data_smithi134_462957_object267:head# 2024-04-25T09:34:12.288 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.305 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1214a567:::benchmark_data_smithi134_468469_object13:head# 2024-04-25T09:34:12.305 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.321 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1255ae71:::benchmark_data_smithi134_462957_object123:head# 2024-04-25T09:34:12.321 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.338 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:131e122e:::benchmark_data_smithi134_468469_object87:head# 2024-04-25T09:34:12.338 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.355 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:134c0c2c:::benchmark_data_smithi134_462957_object140:head# 2024-04-25T09:34:12.355 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.371 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:136d3c79:::benchmark_data_smithi134_462957_object216:head# 2024-04-25T09:34:12.371 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.388 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:14c28473:::benchmark_data_smithi134_462957_object143:head# 2024-04-25T09:34:12.388 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.405 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:14d46dea:::benchmark_data_smithi134_462957_object322:head# 2024-04-25T09:34:12.405 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.421 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:151932bc:::benchmark_data_smithi134_462957_object127:head# 2024-04-25T09:34:12.421 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.438 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:15788a9d:::benchmark_data_smithi134_462957_object215:head# 2024-04-25T09:34:12.438 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.455 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1607bd5a:::benchmark_data_smithi134_462957_object24:head# 2024-04-25T09:34:12.455 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.471 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:16357f0a:::benchmark_data_smithi134_462957_object16:head# 2024-04-25T09:34:12.471 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.488 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:16baa595:::benchmark_data_smithi134_462957_object309:head# 2024-04-25T09:34:12.489 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.505 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:177ef5cc:::benchmark_data_smithi134_468469_object41:head# 2024-04-25T09:34:12.505 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.521 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:18e81c1e:::benchmark_data_smithi134_468469_object52:head# 2024-04-25T09:34:12.521 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.538 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1965afdf:::benchmark_data_smithi134_462957_object89:head# 2024-04-25T09:34:12.538 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.555 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1affb4ca:::benchmark_data_smithi134_462957_object62:head# 2024-04-25T09:34:12.555 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.571 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1b230e12:::benchmark_data_smithi134_462957_object118:head# 2024-04-25T09:34:12.572 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.588 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1bd04fc7:::benchmark_data_smithi134_462957_object188:head# 2024-04-25T09:34:12.588 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.605 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1c79513d:::benchmark_data_smithi134_468469_object117:head# 2024-04-25T09:34:12.605 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.621 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1d14325e:::benchmark_data_smithi134_462957_object41:head# 2024-04-25T09:34:12.622 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.638 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1d1e4cdb:::benchmark_data_smithi134_468469_object45:head# 2024-04-25T09:34:12.638 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.655 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1d8f10c1:::benchmark_data_smithi134_462957_object86:head# 2024-04-25T09:34:12.655 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.672 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1da2d015:::benchmark_data_smithi134_462957_object64:head# 2024-04-25T09:34:12.672 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.688 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1dad9b9c:::benchmark_data_smithi134_462957_object257:head# 2024-04-25T09:34:12.689 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.705 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1efb0d17:::benchmark_data_smithi134_468469_object17:head# 2024-04-25T09:34:12.705 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.722 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1fb02fa0:::benchmark_data_smithi134_462957_object193:head# 2024-04-25T09:34:12.722 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.738 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:203c2195:::benchmark_data_smithi134_462957_object254:head# 2024-04-25T09:34:12.738 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.755 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:22302b8d:::benchmark_data_smithi134_462957_object80:head# 2024-04-25T09:34:12.755 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.772 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:226a74e1:::benchmark_data_smithi134_468469_object138:head# 2024-04-25T09:34:12.772 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.788 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:228f6133:::benchmark_data_smithi134_462957_object74:head# 2024-04-25T09:34:12.789 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.805 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:234eb4d4:::benchmark_data_smithi134_468469_object69:head# 2024-04-25T09:34:12.805 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.822 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:23e9d7b5:::benchmark_data_smithi134_462957_object245:head# 2024-04-25T09:34:12.822 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.856 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:23fa9a3a:::benchmark_data_smithi134_468469_object50:head# 2024-04-25T09:34:12.857 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.932 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2426c803:::benchmark_data_smithi134_462957_object290:head# 2024-04-25T09:34:12.932 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.960 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:244ef85c:::benchmark_data_smithi134_462957_object170:head# 2024-04-25T09:34:12.961 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:12.999 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2513bbe2:::benchmark_data_smithi134_462957_object207:head# 2024-04-25T09:34:12.999 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.041 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2568f8b5:::benchmark_data_smithi134_462957_object8:head# 2024-04-25T09:34:13.041 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.055 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:260f1533:::benchmark_data_smithi134_462957_object191:head# 2024-04-25T09:34:13.055 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.133 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2671aca6:::benchmark_data_smithi134_468469_object15:head# 2024-04-25T09:34:13.133 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.182 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:27170e69:::benchmark_data_smithi134_462957_object34:head# 2024-04-25T09:34:13.182 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.214 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:275fd7c3:::benchmark_data_smithi134_462957_object225:head# 2024-04-25T09:34:13.214 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.230 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:28ef7a99:::benchmark_data_smithi134_462957_object162:head# 2024-04-25T09:34:13.230 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.247 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2a5b63ae:::benchmark_data_smithi134_468469_object77:head# 2024-04-25T09:34:13.247 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.264 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2b3177c6:::benchmark_data_smithi134_462957_object298:head# 2024-04-25T09:34:13.264 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.280 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2b9356c2:::benchmark_data_smithi134_462957_object270:head# 2024-04-25T09:34:13.280 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.297 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2cb634ef:::benchmark_data_smithi134_462957_object297:head# 2024-04-25T09:34:13.297 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.314 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2ccb43ac:::benchmark_data_smithi134_462957_object28:head# 2024-04-25T09:34:13.314 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.330 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2cf40efb:::benchmark_data_smithi134_462957_object289:head# 2024-04-25T09:34:13.331 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.347 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2d9f151f:::benchmark_data_smithi134_462957_object73:head# 2024-04-25T09:34:13.347 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.364 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2dfb304e:::benchmark_data_smithi134_462957_object138:head# 2024-04-25T09:34:13.364 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.380 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2e76756e:::benchmark_data_smithi134_462957_object174:head# 2024-04-25T09:34:13.380 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.397 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2e90f24f:::benchmark_data_smithi134_468469_object124:head# 2024-04-25T09:34:13.397 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.414 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2ed12146:::benchmark_data_smithi134_468469_object44:head# 2024-04-25T09:34:13.414 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.430 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2f2a62da:::benchmark_data_smithi134_462957_object230:head# 2024-04-25T09:34:13.430 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.447 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2f790bda:::benchmark_data_smithi134_468469_object8:head# 2024-04-25T09:34:13.447 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2f954ef6:::benchmark_data_smithi134_462957_object120:head# 2024-04-25T09:34:13.464 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.481 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:302de82f:::benchmark_data_smithi134_462957_object268:head# 2024-04-25T09:34:13.481 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.497 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:30d87a29:::benchmark_data_smithi134_468469_object29:head# 2024-04-25T09:34:13.497 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.514 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:314aacf8:::benchmark_data_smithi134_462957_object98:head# 2024-04-25T09:34:13.514 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.531 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:31df473b:::benchmark_data_smithi134_468469_object43:head# 2024-04-25T09:34:13.531 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.547 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:328e347c:::benchmark_data_smithi134_462957_object203:head# 2024-04-25T09:34:13.547 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.564 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:33056cad:::benchmark_data_smithi134_462957_object248:head# 2024-04-25T09:34:13.564 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.581 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3342be62:::benchmark_data_smithi134_462957_object158:head# 2024-04-25T09:34:13.581 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.597 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:33c19275:::benchmark_data_smithi134_462957_object43:head# 2024-04-25T09:34:13.597 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.614 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:34187219:::benchmark_data_smithi134_462957_object294:head# 2024-04-25T09:34:13.614 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.658 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:344b976e:::benchmark_data_smithi134_462957_object231:head# 2024-04-25T09:34:13.658 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.706 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:35331aa1:::benchmark_data_smithi134_468469_object72:head# 2024-04-25T09:34:13.706 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.747 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:358811ac:::benchmark_data_smithi134_462957_object142:head# 2024-04-25T09:34:13.748 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.756 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:35c66d5e:::benchmark_data_smithi134_468469_object75:head# 2024-04-25T09:34:13.756 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.772 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:361e89fa:::benchmark_data_smithi134_462957_object308:head# 2024-04-25T09:34:13.773 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.789 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:361f3750:::benchmark_data_smithi134_468469_object109:head# 2024-04-25T09:34:13.789 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.806 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:369c019f:::benchmark_data_smithi134_462957_object313:head# 2024-04-25T09:34:13.806 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.822 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:36a6d606:::benchmark_data_smithi134_468469_object73:head# 2024-04-25T09:34:13.823 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.839 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:382b9d8b:::benchmark_data_smithi134_468469_object127:head# 2024-04-25T09:34:13.840 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.856 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:39249c23:::benchmark_data_smithi134_462957_object161:head# 2024-04-25T09:34:13.856 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.873 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3a8ee0cf:::benchmark_data_smithi134_462957_object151:head# 2024-04-25T09:34:13.873 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.889 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3b3824a5:::benchmark_data_smithi134_468469_object111:head# 2024-04-25T09:34:13.889 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.906 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3b9ee75a:::benchmark_data_smithi134_462957_object326:head# 2024-04-25T09:34:13.906 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.922 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3d341765:::benchmark_data_smithi134_468469_object62:head# 2024-04-25T09:34:13.923 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.939 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3e5f4585:::benchmark_data_smithi134_462957_object47:head# 2024-04-25T09:34:13.939 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.956 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3ed5c4c3:::benchmark_data_smithi134_462957_object205:head# 2024-04-25T09:34:13.956 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.972 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3f3910d9:::benchmark_data_smithi134_468469_object98:head# 2024-04-25T09:34:13.973 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.989 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3fc6ca95:::benchmark_data_smithi134_462957_object12:head# 2024-04-25T09:34:13.989 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:13.997 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:40470fd1:::benchmark_data_smithi134_462957_object17:head# 2024-04-25T09:34:13.998 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.014 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:417961ce:::benchmark_data_smithi134_462957_object319:head# 2024-04-25T09:34:14.014 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.031 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:418ec0ee:::benchmark_data_smithi134_462957_object149:head# 2024-04-25T09:34:14.031 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.048 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:419edc7e:::benchmark_data_smithi134_468469_object26:head# 2024-04-25T09:34:14.048 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.108 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:422d92f6:::benchmark_data_smithi134_462957_object272:head# 2024-04-25T09:34:14.109 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.145 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:42b97c1d:::benchmark_data_smithi134_468469_object112:head# 2024-04-25T09:34:14.146 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.184 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:42c245f2:::benchmark_data_smithi134_468469_object99:head# 2024-04-25T09:34:14.184 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.198 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:435e2086:::benchmark_data_smithi134_462957_object171:head# 2024-04-25T09:34:14.198 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.214 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:441b2425:::benchmark_data_smithi134_468469_object136:head# 2024-04-25T09:34:14.215 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.223 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:444a49fd:::benchmark_data_smithi134_462957_object328:head# 2024-04-25T09:34:14.223 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.239 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:44a8dd41:::benchmark_data_smithi134_462957_object3:head# 2024-04-25T09:34:14.240 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.256 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:467046b5:::benchmark_data_smithi134_462957_object265:head# 2024-04-25T09:34:14.257 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.273 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4672b82d:::benchmark_data_smithi134_462957_object295:head# 2024-04-25T09:34:14.273 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.289 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47196e71:::benchmark_data_smithi134_462957_object269:head# 2024-04-25T09:34:14.290 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.298 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:478e238f:::benchmark_data_smithi134_462957_object213:head# 2024-04-25T09:34:14.298 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.314 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47a3ca01:::benchmark_data_smithi134_468469_object90:head# 2024-04-25T09:34:14.315 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.331 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47e25b7f:::benchmark_data_smithi134_462957_object249:head# 2024-04-25T09:34:14.332 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.348 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47f5dd1f:::benchmark_data_smithi134_462957_object122:head# 2024-04-25T09:34:14.348 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.365 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:49a59676:::benchmark_data_smithi134_468469_object4:head# 2024-04-25T09:34:14.365 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.415 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:49e180d1:::benchmark_data_smithi134_462957_object88:head# 2024-04-25T09:34:14.415 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.424 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4a676de3:::benchmark_data_smithi134_462957_object263:head# 2024-04-25T09:34:14.425 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.431 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4a88808c:::benchmark_data_smithi134_468469_object80:head# 2024-04-25T09:34:14.432 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.448 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4b79b182:::benchmark_data_smithi134_462957_object137:head# 2024-04-25T09:34:14.448 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.456 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4c691a21:::benchmark_data_smithi134_468469_object89:head# 2024-04-25T09:34:14.457 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.465 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4cb6f1b7:::benchmark_data_smithi134_462957_object78:head# 2024-04-25T09:34:14.465 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.473 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4e62c3c1:::benchmark_data_smithi134_462957_object53:head# 2024-04-25T09:34:14.473 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.481 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4e981d63:::benchmark_data_smithi134_462957_object227:head# 2024-04-25T09:34:14.482 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.490 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4fda476d:::benchmark_data_smithi134_462957_object194:head# 2024-04-25T09:34:14.490 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.498 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:504c6844:::benchmark_data_smithi134_462957_object94:head# 2024-04-25T09:34:14.498 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.507 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:504ee13c:::benchmark_data_smithi134_462957_object93:head# 2024-04-25T09:34:14.507 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.515 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:510547ae:::benchmark_data_smithi134_468469_object46:head# 2024-04-25T09:34:14.515 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.523 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:511d8705:::benchmark_data_smithi134_462957_object271:head# 2024-04-25T09:34:14.524 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.532 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:51b089df:::benchmark_data_smithi134_462957_object273:head# 2024-04-25T09:34:14.532 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.540 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5266c200:::benchmark_data_smithi134_462957_object32:head# 2024-04-25T09:34:14.540 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.548 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:542d36c1:::benchmark_data_smithi134_462957_object4:head# 2024-04-25T09:34:14.548 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.557 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:544671af:::benchmark_data_smithi134_462957_object172:head# 2024-04-25T09:34:14.557 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.565 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:549eba21:::benchmark_data_smithi134_462957_object133:head# 2024-04-25T09:34:14.565 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.573 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:556d8a79:::benchmark_data_smithi134_462957_object214:head# 2024-04-25T09:34:14.573 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.582 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:556e8b3c:::benchmark_data_smithi134_462957_object96:head# 2024-04-25T09:34:14.582 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.590 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:55950821:::benchmark_data_smithi134_462957_object141:head# 2024-04-25T09:34:14.590 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.598 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:55fef97d:::benchmark_data_smithi134_468469_object38:head# 2024-04-25T09:34:14.598 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.607 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:560a6861:::benchmark_data_smithi134_468469_object131:head# 2024-04-25T09:34:14.607 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.615 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:575ccb6b:::benchmark_data_smithi134_462957_object280:head# 2024-04-25T09:34:14.615 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.623 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:57a020db:::benchmark_data_smithi134_462957_object286:head# 2024-04-25T09:34:14.623 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.632 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5853733d:::benchmark_data_smithi134_468469_object119:head# 2024-04-25T09:34:14.632 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.640 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:58587487:::benchmark_data_smithi134_462957_object51:head# 2024-04-25T09:34:14.640 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.648 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5b1ef10f:::benchmark_data_smithi134_468469_object132:head# 2024-04-25T09:34:14.648 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.657 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5b32c1cc:::benchmark_data_smithi134_468469_object7:head# 2024-04-25T09:34:14.657 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.665 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5c373486:::benchmark_data_smithi134_462957_object14:head# 2024-04-25T09:34:14.665 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.674 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5d0c53ae:::benchmark_data_smithi134_468469_object123:head# 2024-04-25T09:34:14.674 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.682 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5d23a64b:::benchmark_data_smithi134_468469_object133:head# 2024-04-25T09:34:14.682 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.690 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5dada283:::benchmark_data_smithi134_462957_object77:head# 2024-04-25T09:34:14.690 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.699 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5e2dac6b:::benchmark_data_smithi134_468469_object81:head# 2024-04-25T09:34:14.699 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.707 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5e48ae44:::benchmark_data_smithi134_462957_object333:head# 2024-04-25T09:34:14.707 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.715 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5e9f3007:::benchmark_data_smithi134_468469_object1:head# 2024-04-25T09:34:14.715 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.724 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6066d321:::benchmark_data_smithi134_468469_object126:head# 2024-04-25T09:34:14.724 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.732 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:60d506d0:::benchmark_data_smithi134_462957_object111:head# 2024-04-25T09:34:14.732 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.740 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:610e8400:::benchmark_data_smithi134_468469_object97:head# 2024-04-25T09:34:14.741 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.749 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:61342236:::benchmark_data_smithi134_462957_object301:head# 2024-04-25T09:34:14.749 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.757 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:61c961a6:::benchmark_data_smithi134_468469_object61:head# 2024-04-25T09:34:14.757 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.765 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:61d1b2a4:::benchmark_data_smithi134_462957_object179:head# 2024-04-25T09:34:14.766 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.774 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:621d43c5:::benchmark_data_smithi134_462957_object102:head# 2024-04-25T09:34:14.774 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.782 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:62545d41:::benchmark_data_smithi134_462957_object318:head# 2024-04-25T09:34:14.782 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.790 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:632309d1:::benchmark_data_smithi134_468469_object0:head# 2024-04-25T09:34:14.791 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.807 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:632b2ff2:::benchmark_data_smithi134_462957_object113:head# 2024-04-25T09:34:14.808 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.824 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:63effe0b:::benchmark_data_smithi134_462957_object1:head# 2024-04-25T09:34:14.824 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.841 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:63f38e63:::benchmark_data_smithi134_462957_object314:head# 2024-04-25T09:34:14.841 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.858 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:645caf70:::benchmark_data_smithi134_462957_object9:head# 2024-04-25T09:34:14.858 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.874 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:64cb656a:::benchmark_data_smithi134_462957_object38:head# 2024-04-25T09:34:14.875 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.891 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:651d1c11:::benchmark_data_smithi134_462957_object219:head# 2024-04-25T09:34:14.891 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.908 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:65698cf1:::benchmark_data_smithi134_462957_object315:head# 2024-04-25T09:34:14.908 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.924 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:660b1875:::benchmark_data_smithi134_462957_object153:head# 2024-04-25T09:34:14.924 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.941 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6631758f:::benchmark_data_smithi134_468469_object66:head# 2024-04-25T09:34:14.941 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.958 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:66869d35:::benchmark_data_smithi134_468469_object102:head# 2024-04-25T09:34:14.958 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.974 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:66e4bb20:::benchmark_data_smithi134_462957_object259:head# 2024-04-25T09:34:14.975 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:14.991 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:670b93a5:::benchmark_data_smithi134_468469_object85:head# 2024-04-25T09:34:14.991 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.008 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67406b53:::benchmark_data_smithi134_462957_object168:head# 2024-04-25T09:34:15.008 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.024 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6753f201:::benchmark_data_smithi134_462957_object19:head# 2024-04-25T09:34:15.024 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.041 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67de4446:::benchmark_data_smithi134_462957_object11:head# 2024-04-25T09:34:15.041 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.058 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68078f8b:::benchmark_data_smithi134_468469_object129:head# 2024-04-25T09:34:15.058 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.075 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68431e33:::benchmark_data_smithi134_462957_object6:head# 2024-04-25T09:34:15.075 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.091 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68da1af8:::benchmark_data_smithi134_462957_object224:head# 2024-04-25T09:34:15.091 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.108 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68e29e1b:::benchmark_data_smithi134_462957_object223:head# 2024-04-25T09:34:15.108 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.150 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:68ff3868:::benchmark_data_smithi134_462957_object104:head# 2024-04-25T09:34:15.150 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.166 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:69444b58:::benchmark_data_smithi134_462957_object236:head# 2024-04-25T09:34:15.166 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.183 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:69ce30ac:::benchmark_data_smithi134_468469_object128:head# 2024-04-25T09:34:15.183 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.200 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6b5b0c73:::benchmark_data_smithi134_468469_object47:head# 2024-04-25T09:34:15.200 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.216 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6bef445e:::benchmark_data_smithi134_462957_object145:head# 2024-04-25T09:34:15.216 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.233 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6c6b0a68:::benchmark_data_smithi134_462957_object106:head# 2024-04-25T09:34:15.233 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.250 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6cb36ab9:::benchmark_data_smithi134_462957_object128:head# 2024-04-25T09:34:15.250 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.266 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6cb830fe:::benchmark_data_smithi134_462957_object281:head# 2024-04-25T09:34:15.266 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.283 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6d038ea7:::benchmark_data_smithi134_462957_object250:head# 2024-04-25T09:34:15.283 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.300 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6dfad7ef:::benchmark_data_smithi134_462957_object253:head# 2024-04-25T09:34:15.300 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.316 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6e8445ed:::benchmark_data_smithi134_462957_object119:head# 2024-04-25T09:34:15.316 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.341 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6ea927e3:::benchmark_data_smithi134_462957_object67:head# 2024-04-25T09:34:15.342 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.367 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6edbe582:::benchmark_data_smithi134_462957_object260:head# 2024-04-25T09:34:15.367 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.375 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6f02c381:::benchmark_data_smithi134_468469_object27:head# 2024-04-25T09:34:15.375 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.383 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6f1d965d:::benchmark_data_smithi134_468469_object101:head# 2024-04-25T09:34:15.383 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.391 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6f922ade:::benchmark_data_smithi134_462957_object136:head# 2024-04-25T09:34:15.392 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.400 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6fe808b0:::benchmark_data_smithi134_462957_object7:head# 2024-04-25T09:34:15.400 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.408 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:71c24ec6:::benchmark_data_smithi134_462957_object35:head# 2024-04-25T09:34:15.408 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.417 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7280df13:::benchmark_data_smithi134_462957_object109:head# 2024-04-25T09:34:15.417 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.425 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:72b74ac7:::benchmark_data_smithi134_462957_object292:head# 2024-04-25T09:34:15.425 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.433 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:72d020e0:::benchmark_data_smithi134_462957_object20:head# 2024-04-25T09:34:15.433 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.442 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:73570709:::benchmark_data_smithi134_462957_object244:head# 2024-04-25T09:34:15.442 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.450 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:73fa9d47:::benchmark_data_smithi134_462957_object0:head# 2024-04-25T09:34:15.450 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.458 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:76387380:::benchmark_data_smithi134_462957_object103:head# 2024-04-25T09:34:15.458 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.467 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:77414e26:::benchmark_data_smithi134_468469_object144:head# 2024-04-25T09:34:15.467 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.475 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:77a61cee:::benchmark_data_smithi134_462957_object237:head# 2024-04-25T09:34:15.475 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.483 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7826dd0b:::benchmark_data_smithi134_462957_object233:head# 2024-04-25T09:34:15.484 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.492 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:787a592c:::benchmark_data_smithi134_468469_object108:head# 2024-04-25T09:34:15.492 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.500 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7958a491:::benchmark_data_smithi134_462957_object232:head# 2024-04-25T09:34:15.500 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.508 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7a5eed6c:::benchmark_data_smithi134_462957_object55:head# 2024-04-25T09:34:15.508 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.517 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7a922587:::benchmark_data_smithi134_468469_object39:head# 2024-04-25T09:34:15.517 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.525 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7bed12e2:::benchmark_data_smithi134_462957_object90:head# 2024-04-25T09:34:15.525 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.533 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7c53f004:::benchmark_data_smithi134_462957_object183:head# 2024-04-25T09:34:15.534 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.542 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7cc751e9:::benchmark_data_smithi134_462957_object150:head# 2024-04-25T09:34:15.542 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.550 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7de067cb:::benchmark_data_smithi134_462957_object148:head# 2024-04-25T09:34:15.550 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.558 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7e10a619:::benchmark_data_smithi134_462957_object296:head# 2024-04-25T09:34:15.559 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.567 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7f2f5709:::benchmark_data_smithi134_462957_object58:head# 2024-04-25T09:34:15.567 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.575 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7f4d9c89:::benchmark_data_smithi134_462957_object209:head# 2024-04-25T09:34:15.575 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.584 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7fa16e9a:::benchmark_data_smithi134_468469_object51:head# 2024-04-25T09:34:15.584 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.592 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7fa509a6:::benchmark_data_smithi134_468469_object120:head# 2024-04-25T09:34:15.592 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.600 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:80b4c8b2:::benchmark_data_smithi134_468469_object22:head# 2024-04-25T09:34:15.600 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.609 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:80f5f423:::benchmark_data_smithi134_462957_object61:head# 2024-04-25T09:34:15.609 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.617 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8136ddfc:::benchmark_data_smithi134_462957_object181:head# 2024-04-25T09:34:15.617 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.642 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:820a3e6e:::benchmark_data_smithi134_468469_object24:head# 2024-04-25T09:34:15.643 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.701 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:822e7d0d:::benchmark_data_smithi134_462957_object36:head# 2024-04-25T09:34:15.701 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.709 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:82344c27:::benchmark_data_smithi134_462957_object239:head# 2024-04-25T09:34:15.709 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.717 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:82ab8645:::benchmark_data_smithi134_462957_object312:head# 2024-04-25T09:34:15.717 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.725 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:83241f30:::benchmark_data_smithi134_462957_object186:head# 2024-04-25T09:34:15.725 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.734 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:833e47e2:::benchmark_data_smithi134_462957_object201:head# 2024-04-25T09:34:15.734 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.742 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:83476594:::benchmark_data_smithi134_468469_object78:head# 2024-04-25T09:34:15.742 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.750 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:838de8c2:::benchmark_data_smithi134_468469_object121:head# 2024-04-25T09:34:15.751 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.759 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:83909346:::benchmark_data_smithi134_462957_object165:head# 2024-04-25T09:34:15.759 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.767 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8486115e:::benchmark_data_smithi134_462957_object134:head# 2024-04-25T09:34:15.767 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.776 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:85663fce:::benchmark_data_smithi134_468469_object104:head# 2024-04-25T09:34:15.776 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.784 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:85d49e01:::benchmark_data_smithi134_462957_object114:head# 2024-04-25T09:34:15.784 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.792 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:86e69dbb:::benchmark_data_smithi134_462957_object334:head# 2024-04-25T09:34:15.792 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.801 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:87789c1f:::benchmark_data_smithi134_462957_object299:head# 2024-04-25T09:34:15.801 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.809 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:87e9176e:::benchmark_data_smithi134_462957_object176:head# 2024-04-25T09:34:15.809 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.817 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:884c4353:::benchmark_data_smithi134_468469_object86:head# 2024-04-25T09:34:15.817 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.826 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:88b81233:::benchmark_data_smithi134_462957_object197:head# 2024-04-25T09:34:15.826 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.834 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8a093cc3:::benchmark_data_smithi134_468469_object14:head# 2024-04-25T09:34:15.834 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.843 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8a1b1ae0:::benchmark_data_smithi134_468469_object91:head# 2024-04-25T09:34:15.843 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.851 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8a55344d:::benchmark_data_smithi134_462957_object163:head# 2024-04-25T09:34:15.851 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.859 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8b35579f:::benchmark_data_smithi134_462957_object101:head# 2024-04-25T09:34:15.859 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.918 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c42d79e:::benchmark_data_smithi134_462957_object285:head# 2024-04-25T09:34:15.918 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.926 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c5cb74b:::benchmark_data_smithi134_462957_object30:head# 2024-04-25T09:34:15.926 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.934 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c777440:::benchmark_data_smithi134_462957_object69:head# 2024-04-25T09:34:15.935 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.943 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8d18392e:::benchmark_data_smithi134_462957_object42:head# 2024-04-25T09:34:15.943 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.951 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8d4a51df:::benchmark_data_smithi134_462957_object202:head# 2024-04-25T09:34:15.951 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.959 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8f9fc238:::benchmark_data_smithi134_462957_object222:head# 2024-04-25T09:34:15.959 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.968 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8fcd40aa:::benchmark_data_smithi134_462957_object13:head# 2024-04-25T09:34:15.968 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.976 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8ffd53b9:::benchmark_data_smithi134_462957_object152:head# 2024-04-25T09:34:15.976 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.984 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9102906f:::benchmark_data_smithi134_468469_object115:head# 2024-04-25T09:34:15.984 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:15.993 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:912ceabf:::benchmark_data_smithi134_462957_object291:head# 2024-04-25T09:34:15.993 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.001 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:913e15cc:::benchmark_data_smithi134_462957_object283:head# 2024-04-25T09:34:16.001 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.009 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:92bf16d1:::benchmark_data_smithi134_462957_object195:head# 2024-04-25T09:34:16.009 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.018 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:92ed1f0f:::benchmark_data_smithi134_462957_object65:head# 2024-04-25T09:34:16.018 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.026 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:93c080bd:::benchmark_data_smithi134_468469_object76:head# 2024-04-25T09:34:16.026 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.034 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9417108d:::benchmark_data_smithi134_462957_object261:head# 2024-04-25T09:34:16.034 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.043 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:94a46faa:::benchmark_data_smithi134_468469_object32:head# 2024-04-25T09:34:16.043 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.069 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:954cc32c:::benchmark_data_smithi134_462957_object139:head# 2024-04-25T09:34:16.069 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.126 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9632b147:::benchmark_data_smithi134_462957_object154:head# 2024-04-25T09:34:16.126 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.168 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:96994074:::benchmark_data_smithi134_468469_object5:head# 2024-04-25T09:34:16.168 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.210 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9726b3f5:::benchmark_data_smithi134_462957_object135:head# 2024-04-25T09:34:16.210 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.219 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:97aa80c9:::benchmark_data_smithi134_462957_object210:head# 2024-04-25T09:34:16.219 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.226 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:98039778:::benchmark_data_smithi134_468469_object110:head# 2024-04-25T09:34:16.226 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.235 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9870b61d:::benchmark_data_smithi134_462957_object50:head# 2024-04-25T09:34:16.235 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.243 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9a3c902a:::benchmark_data_smithi134_462957_object66:head# 2024-04-25T09:34:16.243 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.251 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9a63a41b:::benchmark_data_smithi134_462957_object59:head# 2024-04-25T09:34:16.251 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.260 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9a6d1805:::benchmark_data_smithi134_468469_object141:head# 2024-04-25T09:34:16.260 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.268 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9af4c21c:::benchmark_data_smithi134_462957_object99:head# 2024-04-25T09:34:16.268 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.277 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9c02c888:::benchmark_data_smithi134_462957_object100:head# 2024-04-25T09:34:16.277 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.285 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9c7daede:::benchmark_data_smithi134_468469_object71:head# 2024-04-25T09:34:16.285 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.293 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9d244c8f:::benchmark_data_smithi134_468469_object18:head# 2024-04-25T09:34:16.293 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.302 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9d740ac7:::benchmark_data_smithi134_462957_object156:head# 2024-04-25T09:34:16.302 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.310 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9d9951df:::benchmark_data_smithi134_468469_object9:head# 2024-04-25T09:34:16.310 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.318 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e32be9a:::benchmark_data_smithi134_462957_object54:head# 2024-04-25T09:34:16.318 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.326 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e6056cd:::benchmark_data_smithi134_462957_object164:head# 2024-04-25T09:34:16.327 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.335 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e878f34:::benchmark_data_smithi134_468469_object54:head# 2024-04-25T09:34:16.335 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.343 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9ebf06db:::benchmark_data_smithi134_462957_object284:head# 2024-04-25T09:34:16.343 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.352 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9ee7691c:::benchmark_data_smithi134_468469_object65:head# 2024-04-25T09:34:16.352 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.360 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a045e8d8:::benchmark_data_smithi134_462957_object234:head# 2024-04-25T09:34:16.360 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.368 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a08b0a4d:::benchmark_data_smithi134_462957_object241:head# 2024-04-25T09:34:16.368 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.377 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a0c022f8:::benchmark_data_smithi134_462957_object226:head# 2024-04-25T09:34:16.377 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.385 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a1658627:::benchmark_data_smithi134_462957_object327:head# 2024-04-25T09:34:16.385 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.393 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a1bb4fbf:::benchmark_data_smithi134_462957_object198:head# 2024-04-25T09:34:16.393 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.402 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a2b4a5bd:::benchmark_data_smithi134_462957_object27:head# 2024-04-25T09:34:16.402 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.410 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a3329f50:::benchmark_data_smithi134_468469_object107:head# 2024-04-25T09:34:16.410 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.418 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a50fb507:::benchmark_data_smithi134_468469_object118:head# 2024-04-25T09:34:16.418 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.427 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a5409620:::benchmark_data_smithi134_468469_object60:head# 2024-04-25T09:34:16.427 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.435 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a5ca7c00:::benchmark_data_smithi134_462957_object72:head# 2024-04-25T09:34:16.435 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.443 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a5f62c63:::benchmark_data_smithi134_462957_object316:head# 2024-04-25T09:34:16.444 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.452 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a63f834f:::benchmark_data_smithi134_462957_object157:head# 2024-04-25T09:34:16.452 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.460 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a715009b:::benchmark_data_smithi134_462957_object169:head# 2024-04-25T09:34:16.460 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.468 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a85dbb56:::benchmark_data_smithi134_462957_object307:head# 2024-04-25T09:34:16.469 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.477 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a8767535:::benchmark_data_smithi134_468469_object92:head# 2024-04-25T09:34:16.477 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.485 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a8d1cd5f:::benchmark_data_smithi134_462957_object107:head# 2024-04-25T09:34:16.485 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.494 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a9621e84:::benchmark_data_smithi134_462957_object303:head# 2024-04-25T09:34:16.494 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.502 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a9a511c7:::benchmark_data_smithi134_462957_object321:head# 2024-04-25T09:34:16.502 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.510 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a9a8cefb:::benchmark_data_smithi134_468469_object36:head# 2024-04-25T09:34:16.510 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.519 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:aa4d3c89:::benchmark_data_smithi134_468469_object34:head# 2024-04-25T09:34:16.519 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.527 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ab3c651b:::benchmark_data_smithi134_468469_object58:head# 2024-04-25T09:34:16.527 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.535 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ab6edda1:::benchmark_data_smithi134_462957_object212:head# 2024-04-25T09:34:16.535 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.544 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:aba338b0:::benchmark_data_smithi134_462957_object184:head# 2024-04-25T09:34:16.544 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.552 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:abdd83aa:::benchmark_data_smithi134_462957_object85:head# 2024-04-25T09:34:16.552 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.560 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:abf2d6a2:::benchmark_data_smithi134_468469_object48:head# 2024-04-25T09:34:16.560 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.569 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ac85576b:::benchmark_data_smithi134_462957_object282:head# 2024-04-25T09:34:16.569 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.577 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ad4a0ace:::benchmark_data_smithi134_468469_object116:head# 2024-04-25T09:34:16.577 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.602 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ae25ce97:::benchmark_data_smithi134_462957_object75:head# 2024-04-25T09:34:16.602 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.611 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:aeaa246a:::benchmark_data_smithi134_468469_object88:head# 2024-04-25T09:34:16.611 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.619 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b1f8b3b5:::benchmark_data_smithi134_462957_object160:head# 2024-04-25T09:34:16.619 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.627 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b249419b:::benchmark_data_smithi134_462957_object275:head# 2024-04-25T09:34:16.627 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.635 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b289940b:::benchmark_data_smithi134_462957_object26:head# 2024-04-25T09:34:16.636 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.644 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b29d1223:::benchmark_data_smithi134_462957_object167:head# 2024-04-25T09:34:16.644 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.652 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b4ae5688:::benchmark_data_smithi134_462957_object182:head# 2024-04-25T09:34:16.652 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.661 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b5565581:::benchmark_data_smithi134_462957_object22:head# 2024-04-25T09:34:16.661 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.669 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b56627a7:::benchmark_data_smithi134_462957_object108:head# 2024-04-25T09:34:16.669 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.677 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b62c98b7:::benchmark_data_smithi134_468469_object137:head# 2024-04-25T09:34:16.677 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.686 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b6ff86cd:::benchmark_data_smithi134_462957_object166:head# 2024-04-25T09:34:16.686 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.694 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b7edf419:::benchmark_data_smithi134_462957_object84:head# 2024-04-25T09:34:16.694 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.732 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:34:16.733 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.752 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b8f154ff:::benchmark_data_smithi134_462957_object206:head# 2024-04-25T09:34:16.752 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.761 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b920d5ca:::benchmark_data_smithi134_468469_object94:head# 2024-04-25T09:34:16.761 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.769 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b93070b3:::benchmark_data_smithi134_468469_object55:head# 2024-04-25T09:34:16.769 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.777 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b93b7738:::benchmark_data_smithi134_462957_object92:head# 2024-04-25T09:34:16.778 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.786 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b997874c:::benchmark_data_smithi134_462957_object320:head# 2024-04-25T09:34:16.786 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.794 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ba34c80e:::benchmark_data_smithi134_462957_object29:head# 2024-04-25T09:34:16.794 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.802 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ba9726b3:::benchmark_data_smithi134_468469_object95:head# 2024-04-25T09:34:16.803 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.811 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bb51ab09:::benchmark_data_smithi134_462957_object246:head# 2024-04-25T09:34:16.811 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.819 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bb9ed292:::benchmark_data_smithi134_462957_object44:head# 2024-04-25T09:34:16.819 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.828 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bc0fe2eb:::benchmark_data_smithi134_462957_object105:head# 2024-04-25T09:34:16.828 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.836 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bd33c13c:::benchmark_data_smithi134_462957_object121:head# 2024-04-25T09:34:16.836 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.844 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bd52ec16:::benchmark_data_smithi134_462957_object68:head# 2024-04-25T09:34:16.844 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.853 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bd71e327:::benchmark_data_smithi134_468469_object10:head# 2024-04-25T09:34:16.853 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.861 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:be19e778:::benchmark_data_smithi134_462957_object238:head# 2024-04-25T09:34:16.861 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.869 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bf64a100:::benchmark_data_smithi134_462957_object83:head# 2024-04-25T09:34:16.869 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.878 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c03f464f:::benchmark_data_smithi134_462957_object18:head# 2024-04-25T09:34:16.878 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.886 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c0ec0650:::benchmark_data_smithi134_468469_object93:head# 2024-04-25T09:34:16.886 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.894 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c0f283ce:::benchmark_data_smithi134_462957_object124:head# 2024-04-25T09:34:16.895 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.903 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c16c7b78:::benchmark_data_smithi134_462957_object2:head# 2024-04-25T09:34:16.903 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.914 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c184615c:::benchmark_data_smithi134_468469_object28:head# 2024-04-25T09:34:16.914 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.919 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c18f6438:::benchmark_data_smithi134_462957_object220:head# 2024-04-25T09:34:16.920 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.928 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c244846a:::benchmark_data_smithi134_462957_object329:head# 2024-04-25T09:34:16.928 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.936 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c274bcc9:::benchmark_data_smithi134_462957_object332:head# 2024-04-25T09:34:16.936 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.945 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c2e9f6b4:::benchmark_data_smithi134_462957_object330:head# 2024-04-25T09:34:16.945 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.953 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c35cfddf:::benchmark_data_smithi134_462957_object200:head# 2024-04-25T09:34:16.953 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.961 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c35d1837:::benchmark_data_smithi134_468469_object103:head# 2024-04-25T09:34:16.961 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.969 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c47b78d0:::benchmark_data_smithi134_462957_object117:head# 2024-04-25T09:34:16.971 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.978 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c6365174:::benchmark_data_smithi134_468469_object21:head# 2024-04-25T09:34:16.978 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.986 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c70a7496:::benchmark_data_smithi134_462957_object60:head# 2024-04-25T09:34:16.986 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:16.995 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c73308dd:::benchmark_data_smithi134_462957_object258:head# 2024-04-25T09:34:16.995 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.003 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c74b145d:::benchmark_data_smithi134_462957_object146:head# 2024-04-25T09:34:17.003 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.011 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c7d37ef1:::benchmark_data_smithi134_462957_object125:head# 2024-04-25T09:34:17.011 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.020 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c7e782f1:::benchmark_data_smithi134_468469_object84:head# 2024-04-25T09:34:17.020 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.028 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c84d6c63:::benchmark_data_smithi134_462957_object221:head# 2024-04-25T09:34:17.029 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.037 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c857d305:::benchmark_data_smithi134_468469_object96:head# 2024-04-25T09:34:17.037 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.045 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c8d637f7:::benchmark_data_smithi134_462957_object306:head# 2024-04-25T09:34:17.045 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.053 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c9e347b1:::benchmark_data_smithi134_462957_object242:head# 2024-04-25T09:34:17.053 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.062 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ca1ed64b:::benchmark_data_smithi134_468469_object135:head# 2024-04-25T09:34:17.062 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.070 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ca2673ae:::benchmark_data_smithi134_468469_object125:head# 2024-04-25T09:34:17.070 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.078 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ca46b863:::benchmark_data_smithi134_462957_object57:head# 2024-04-25T09:34:17.078 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.086 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ca795a74:::benchmark_data_smithi134_462957_object81:head# 2024-04-25T09:34:17.087 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.095 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cae41921:::benchmark_data_smithi134_468469_object106:head# 2024-04-25T09:34:17.095 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.103 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cc49d25d:::benchmark_data_smithi134_462957_object144:head# 2024-04-25T09:34:17.103 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.111 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cc623ab1:::benchmark_data_smithi134_468469_object25:head# 2024-04-25T09:34:17.112 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.120 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ccd8cbfe:::benchmark_data_smithi134_462957_object5:head# 2024-04-25T09:34:17.120 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.128 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cce05002:::benchmark_data_smithi134_462957_object130:head# 2024-04-25T09:34:17.129 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.137 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ccfcf8fa:::benchmark_data_smithi134_468469_object70:head# 2024-04-25T09:34:17.137 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.145 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cd3e8f8e:::benchmark_data_smithi134_468469_object53:head# 2024-04-25T09:34:17.145 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.153 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cd79c5f8:::benchmark_data_smithi134_462957_object76:head# 2024-04-25T09:34:17.154 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.162 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cdb281d5:::benchmark_data_smithi134_462957_object46:head# 2024-04-25T09:34:17.162 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.170 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cdc69e54:::benchmark_data_smithi134_462957_object252:head# 2024-04-25T09:34:17.170 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.212 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf2a4062:::benchmark_data_smithi134_462957_object199:head# 2024-04-25T09:34:17.212 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.220 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf35812c:::benchmark_data_smithi134_462957_object243:head# 2024-04-25T09:34:17.220 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.229 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf693ea1:::benchmark_data_smithi134_462957_object147:head# 2024-04-25T09:34:17.229 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.237 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d0dad50c:::benchmark_data_smithi134_462957_object97:head# 2024-04-25T09:34:17.237 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.245 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d1761cb1:::benchmark_data_smithi134_462957_object251:head# 2024-04-25T09:34:17.245 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.253 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d1b7fc50:::benchmark_data_smithi134_462957_object211:head# 2024-04-25T09:34:17.254 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.262 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d2bd3105:::benchmark_data_smithi134_462957_object277:head# 2024-04-25T09:34:17.262 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.270 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d3201ed2:::benchmark_data_smithi134_462957_object45:head# 2024-04-25T09:34:17.270 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.279 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d3f96c8d:::benchmark_data_smithi134_468469_object140:head# 2024-04-25T09:34:17.279 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.287 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d423debd:::benchmark_data_smithi134_462957_object115:head# 2024-04-25T09:34:17.287 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.295 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d4924696:::benchmark_data_smithi134_468469_object35:head# 2024-04-25T09:34:17.295 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.304 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d4c4f287:::benchmark_data_smithi134_462957_object21:head# 2024-04-25T09:34:17.304 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.312 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d50a78a1:::benchmark_data_smithi134_462957_object25:head# 2024-04-25T09:34:17.312 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.320 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d5af3bd1:::benchmark_data_smithi134_462957_object180:head# 2024-04-25T09:34:17.320 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.329 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d5ef9c20:::benchmark_data_smithi134_462957_object70:head# 2024-04-25T09:34:17.329 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.337 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6221551:::benchmark_data_smithi134_468469_object74:head# 2024-04-25T09:34:17.337 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.345 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d65dc781:::benchmark_data_smithi134_462957_object323:head# 2024-04-25T09:34:17.345 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.354 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6872c2f:::benchmark_data_smithi134_462957_object192:head# 2024-04-25T09:34:17.354 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.363 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6cbb661:::benchmark_data_smithi134_462957_object279:head# 2024-04-25T09:34:17.363 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.371 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d6d8a7de:::benchmark_data_smithi134_462957_object175:head# 2024-04-25T09:34:17.371 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.379 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d9df41a0:::benchmark_data_smithi134_462957_object288:head# 2024-04-25T09:34:17.379 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.387 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dba2aa95:::benchmark_data_smithi134_462957_object31:head# 2024-04-25T09:34:17.387 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.396 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dc540d1b:::benchmark_data_smithi134_462957_object189:head# 2024-04-25T09:34:17.396 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.404 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dc6e92dd:::benchmark_data_smithi134_462957_object159:head# 2024-04-25T09:34:17.404 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.412 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dc96f6c8:::benchmark_data_smithi134_468469_object3:head# 2024-04-25T09:34:17.412 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.421 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dcca6922:::benchmark_data_smithi134_462957_object311:head# 2024-04-25T09:34:17.421 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.429 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd1204a0:::benchmark_data_smithi134_462957_object71:head# 2024-04-25T09:34:17.429 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.437 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd2acaa8:::benchmark_data_smithi134_468469_object2:head# 2024-04-25T09:34:17.437 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.446 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd3e6d82:::benchmark_data_smithi134_462957_object262:head# 2024-04-25T09:34:17.446 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.454 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:df8a97e3:::benchmark_data_smithi134_468469_object33:head# 2024-04-25T09:34:17.454 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.463 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dfc1c8cf:::benchmark_data_smithi134_462957_object229:head# 2024-04-25T09:34:17.463 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.471 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e04321de:::benchmark_data_smithi134_468469_object114:head# 2024-04-25T09:34:17.471 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.479 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e06b8840:::benchmark_data_smithi134_462957_object91:head# 2024-04-25T09:34:17.479 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.488 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e06cab0a:::benchmark_data_smithi134_462957_object23:head# 2024-04-25T09:34:17.488 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.496 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e2819600:::benchmark_data_smithi134_468469_object12:head# 2024-04-25T09:34:17.496 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.504 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e28f6f75:::benchmark_data_smithi134_462957_object300:head# 2024-04-25T09:34:17.504 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.513 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e2be0804:::benchmark_data_smithi134_462957_object235:head# 2024-04-25T09:34:17.513 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.521 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e2c8c0bf:::benchmark_data_smithi134_462957_object40:head# 2024-04-25T09:34:17.521 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.529 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e36a9ada:::benchmark_data_smithi134_468469_object100:head# 2024-04-25T09:34:17.530 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.538 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e3a40c39:::benchmark_data_smithi134_462957_object63:head# 2024-04-25T09:34:17.538 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.546 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e3c5ddbe:::benchmark_data_smithi134_468469_object64:head# 2024-04-25T09:34:17.546 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.554 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e5228945:::benchmark_data_smithi134_462957_object293:head# 2024-04-25T09:34:17.554 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.563 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e5b32a73:::benchmark_data_smithi134_468469_object37:head# 2024-04-25T09:34:17.563 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.571 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e614399c:::benchmark_data_smithi134_462957_object110:head# 2024-04-25T09:34:17.571 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.579 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e65dd3c4:::benchmark_data_smithi134_468469_object130:head# 2024-04-25T09:34:17.579 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.588 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e65f056d:::benchmark_data_smithi134_462957_object196:head# 2024-04-25T09:34:17.588 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.596 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e6a6c722:::benchmark_data_smithi134_462957_object264:head# 2024-04-25T09:34:17.596 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.605 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e74bd302:::benchmark_data_smithi134_462957_object131:head# 2024-04-25T09:34:17.605 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.613 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e795db49:::benchmark_data_smithi134_468469_object42:head# 2024-04-25T09:34:17.613 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.621 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e7aa26a1:::benchmark_data_smithi134_462957_object129:head# 2024-04-25T09:34:17.621 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.668 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e8378195:::benchmark_data_smithi134_462957_object256:head# 2024-04-25T09:34:17.668 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.696 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e8ce525c:::benchmark_data_smithi134_462957_object155:head# 2024-04-25T09:34:17.696 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.704 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e8f5abd5:::benchmark_data_smithi134_462957_object56:head# 2024-04-25T09:34:17.705 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.713 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e93b4e82:::benchmark_data_smithi134_468469_object16:head# 2024-04-25T09:34:17.713 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.746 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e985cf7e:::benchmark_data_smithi134_462957_object287:head# 2024-04-25T09:34:17.746 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.754 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e9f08f42:::benchmark_data_smithi134_462957_object187:head# 2024-04-25T09:34:17.755 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.763 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ea484b6f:::benchmark_data_smithi134_462957_object82:head# 2024-04-25T09:34:17.763 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.771 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:eb7589e9:::benchmark_data_smithi134_468469_object59:head# 2024-04-25T09:34:17.771 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.780 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ed2fe309:::benchmark_data_smithi134_468469_object134:head# 2024-04-25T09:34:17.780 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.788 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ee73ec6d:::benchmark_data_smithi134_462957_object33:head# 2024-04-25T09:34:17.788 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.796 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ee90a495:::benchmark_data_smithi134_468469_object57:head# 2024-04-25T09:34:17.796 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.805 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ef28e1d3:::benchmark_data_smithi134_468469_object31:head# 2024-04-25T09:34:17.805 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.813 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f01022da:::benchmark_data_smithi134_462957_object324:head# 2024-04-25T09:34:17.813 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.821 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f030e4b2:::benchmark_data_smithi134_468469_object67:head# 2024-04-25T09:34:17.821 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.856 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0800d50:::benchmark_data_smithi134_462957_object217:head# 2024-04-25T09:34:17.856 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.871 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0c0db8e:::benchmark_data_smithi134_468469_object19:head# 2024-04-25T09:34:17.872 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.880 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0f00dfc:::benchmark_data_smithi134_462957_object185:head# 2024-04-25T09:34:17.880 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.888 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0f2cea2:::benchmark_data_smithi134_462957_object317:head# 2024-04-25T09:34:17.888 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.910 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f131ea49:::benchmark_data_smithi134_462957_object15:head# 2024-04-25T09:34:17.910 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:17.977 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f24c3070:::benchmark_data_smithi134_468469_object79:head# 2024-04-25T09:34:17.977 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.020 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f3c72ea5:::benchmark_data_smithi134_462957_object173:head# 2024-04-25T09:34:18.020 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.069 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f439d218:::benchmark_data_smithi134_468469_object30:head# 2024-04-25T09:34:18.069 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.152 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f581c1b4:::benchmark_data_smithi134_462957_object218:head# 2024-04-25T09:34:18.153 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.203 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f63f8b4d:::benchmark_data_smithi134_462957_object247:head# 2024-04-25T09:34:18.203 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.286 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f688bb37:::benchmark_data_smithi134_468469_object139:head# 2024-04-25T09:34:18.287 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.353 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f6fa688e:::benchmark_data_smithi134_468469_object142:head# 2024-04-25T09:34:18.353 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.420 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f70159a7:::benchmark_data_smithi134_462957_object208:head# 2024-04-25T09:34:18.423 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.438 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f741ec8b:::benchmark_data_smithi134_462957_object325:head# 2024-04-25T09:34:18.439 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.447 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f7f2b0bd:::benchmark_data_smithi134_462957_object302:head# 2024-04-25T09:34:18.447 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.455 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f849895e:::benchmark_data_smithi134_468469_object11:head# 2024-04-25T09:34:18.455 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.463 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f8a62bfb:::benchmark_data_smithi134_468469_object122:head# 2024-04-25T09:34:18.464 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.472 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f8d2618b:::benchmark_data_smithi134_462957_object255:head# 2024-04-25T09:34:18.472 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.480 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f8dab886:::benchmark_data_smithi134_468469_object82:head# 2024-04-25T09:34:18.480 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.488 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f8f97fba:::benchmark_data_smithi134_462957_object178:head# 2024-04-25T09:34:18.489 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.497 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f9f3f45e:::benchmark_data_smithi134_468469_object143:head# 2024-04-25T09:34:18.497 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.505 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fa5b5d21:::benchmark_data_smithi134_462957_object274:head# 2024-04-25T09:34:18.505 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.513 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fa93c7ab:::benchmark_data_smithi134_468469_object63:head# 2024-04-25T09:34:18.514 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.522 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:faa372e4:::benchmark_data_smithi134_462957_object228:head# 2024-04-25T09:34:18.522 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.530 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fb20bb8f:::benchmark_data_smithi134_462957_object190:head# 2024-04-25T09:34:18.530 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.539 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fb6d87d7:::benchmark_data_smithi134_468469_object105:head# 2024-04-25T09:34:18.539 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.551 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fc414211:::benchmark_data_smithi134_462957_object79:head# 2024-04-25T09:34:18.551 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.555 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fc7f33f8:::benchmark_data_smithi134_462957_object37:head# 2024-04-25T09:34:18.555 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.564 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fe21a7b2:::benchmark_data_smithi134_468469_object68:head# 2024-04-25T09:34:18.564 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.572 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fe933281:::benchmark_data_smithi134_468469_object6:head# 2024-04-25T09:34:18.572 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.580 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fee7f4ff:::benchmark_data_smithi134_462957_object204:head# 2024-04-25T09:34:18.580 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.589 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ff13f1d8:::benchmark_data_smithi134_462957_object49:head# 2024-04-25T09:34:18.589 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.597 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ff758221:::benchmark_data_smithi134_468469_object56:head# 2024-04-25T09:34:18.597 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:34:18.605 INFO:tasks.workunit.client.0.smithi134.stdout:write_pg epoch 224 info 1.0( v 224'316 (9'170,224'316] local-lis/les=23/24 n=482 ec=7/7 lis/c=23/23 les/c/f=24/24/0 sis=23) 2024-04-25T09:34:18.622 INFO:tasks.workunit.client.0.smithi134.stdout:Import successful 2024-04-25T09:34:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:157: TEST_import_after_merge_and_gap: activate_osd td/pg-split-merge 0 2024-04-25T09:34:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:34:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:34:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:34:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:34:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:34:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:34:19.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:19.170 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:34:19.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:34:19.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:34:19.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:34:19.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:34:19.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:34:19.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:34:19.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:34:19.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:34:19.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:34:19.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:34:19.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:34:19.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:34:19.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:34:19.173 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:34:19.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:34:19.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:34:19.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:34:19.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:34:19.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:34:19.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:34:19.207 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:19.220+0000 7fd645daa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:19.209 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:19.224+0000 7fd645daa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:19.211 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:19.224+0000 7fd645daa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:19.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:34:19.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:34:19.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:34:19.598 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:34:19.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:34:19.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:34:19.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:19.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:34:19.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:19.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:19.777 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:19.792+0000 7fd645daa7c0 -1 Falling back to public interface 2024-04-25T09:34:20.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:21.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:21.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:21.047 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:34:21.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:34:21.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:21.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:21.465 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:22.379 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:22.392+0000 7fd645daa7c0 -1 osd.0 228 log_to_monitors true 2024-04-25T09:34:22.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:22.467 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:22.468 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:34:22.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:34:22.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:22.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:23.071 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 231 up_thru 23 down_at 229 last_clean_interval [23,228) [v2:127.0.0.1:6802/3246747064,v1:127.0.0.1:6803/3246747064] [v2:127.0.0.1:6804/3246747064,v1:127.0.0.1:6805/3246747064] exists,up c787144a-5beb-4391-bbea-edfe5c80e442 2024-04-25T09:34:23.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:34:23.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:34:23.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:34:23.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:159: TEST_import_after_merge_and_gap: wait_for_clean 2024-04-25T09:34:23.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:34:23.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:34:23.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:34:23.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:34:23.074 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:34:23.074 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:34:23.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:34:23.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:34:23.074 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:34:23.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:34:23.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:34:23.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:34:23.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:34:23.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:34:23.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:34:23.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:34:23.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:34:23.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:34:23.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:34:23.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=992137445379 2024-04-25T09:34:23.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 992137445379 2024-04-25T09:34:23.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-992137445379' 2024-04-25T09:34:23.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:34:23.860 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-992137445379 2024-04-25T09:34:23.860 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:34:23.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:34:23.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-992137445379 2024-04-25T09:34:23.861 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:34:23.863 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 992137445379 2024-04-25T09:34:23.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=992137445379 2024-04-25T09:34:23.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 992137445379' 2024-04-25T09:34:23.863 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:34:24.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 992137445379 2024-04-25T09:34:24.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:34:25.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:34:25.281 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:34:25.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 992137445379 2024-04-25T09:34:25.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:34:26.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 299 -eq 0 ']' 2024-04-25T09:34:26.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:34:27.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 992137445379 -lt 992137445379 2024-04-25T09:34:27.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:34:27.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:34:27.101 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:34:27.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:34:27.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:34:27.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:34:27.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:34:27.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:34:27.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:34:27.678 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:34:27.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:34:28.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:34:28.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:34:28.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:34:28.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:34:28.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:34:28.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:34:28.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:34:28.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:34:28.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:34:28.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:34:28.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:34:28.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:34:28.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:34:28.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:34:28.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:34:28.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:34:29.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:34:29.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:34:29.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:34:29.021 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:34:29.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:34:29.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:34:29.024 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:34:29.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:34:29.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:34:29.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:34:29.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:34:29.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:34:29.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:34:29.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:34:29.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:34:29.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:29.049 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:34:29.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451985 2024-04-25T09:34:29.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:34:29.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:34:29.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:14: run: for func in $funcs 2024-04-25T09:34:29.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:15: run: setup td/pg-split-merge 2024-04-25T09:34:29.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/pg-split-merge 2024-04-25T09:34:29.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/pg-split-merge 2024-04-25T09:34:29.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:34:29.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:34:29.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:34:29.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:34:29.052 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:34:29.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:34:29.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:34:29.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:34:29.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:34:29.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:34:29.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:34:29.057 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:34:29.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:34:29.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:34:29.059 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:34:29.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:34:29.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:34:29.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:34:29.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:34:29.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:34:29.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:34:29.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:34:29.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:34:29.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:29.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:34:29.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451985 2024-04-25T09:34:29.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:34:29.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:34:29.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/pg-split-merge 2024-04-25T09:34:29.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:34:29.066 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:29.067 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:34:29.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.451985 2024-04-25T09:34:29.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:34:29.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:34:29.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:34:29.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/pg-split-merge 1' TERM HUP INT 2024-04-25T09:34:29.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:16: run: TEST_import_after_split td/pg-split-merge 2024-04-25T09:34:29.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:163: TEST_import_after_split: local dir=td/pg-split-merge 2024-04-25T09:34:29.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:165: TEST_import_after_split: run_mon td/pg-split-merge a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:34:29.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/pg-split-merge 2024-04-25T09:34:29.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:34:29.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:34:29.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:34:29.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/pg-split-merge/a 2024-04-25T09:34:29.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/pg-split-merge/a --run-dir=td/pg-split-merge --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:34:29.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:34:29.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:34:29.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:34:29.534 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:34:29.534 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:29.534 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:34:29.534 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:34:29.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/pg-split-merge/a '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --mon-cluster-log-file=td/pg-split-merge/log --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-04-25T09:34:29.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:34:29.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:34:29.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:34:29.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:34:29.745 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:34:29.747 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:34:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:34:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:34:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:34:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:34:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:34:29.748 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451985/ceph-mon.a.asok 2024-04-25T09:34:29.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:34:29.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451985/ceph-mon.a.asok config get fsid 2024-04-25T09:34:29.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:34:29.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:34:29.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:34:29.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:34:29.866 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:34:29.866 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:34:29.867 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:34:29.867 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:34:29.868 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:34:29.868 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:29.869 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:34:29.869 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.451985/ceph-mon.a.asok 2024-04-25T09:34:29.869 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:34:29.869 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.451985/ceph-mon.a.asok config get mon_host 2024-04-25T09:34:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:166: TEST_import_after_split: run_mgr td/pg-split-merge x 2024-04-25T09:34:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/pg-split-merge 2024-04-25T09:34:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:34:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:34:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:34:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/pg-split-merge/x 2024-04-25T09:34:29.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:34:30.881 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:34:30.882 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:34:30.882 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:34:30.882 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:34:30.882 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:30.883 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:34:30.883 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:34:30.883 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:34:30.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/pg-split-merge/x '--log-file=td/pg-split-merge/$name.log' '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --run-dir=td/pg-split-merge '--pid-file=td/pg-split-merge/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:167: TEST_import_after_split: run_osd td/pg-split-merge 0 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/pg-split-merge 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:34:30.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:34:30.922 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:34:30.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:34:30.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:34:30.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:34:30.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:34:30.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:34:30.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:34:30.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:34:30.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:34:30.923 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:34:30.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=aa1700fc-f35b-4ac9-812c-60d0b8a9b874 2024-04-25T09:34:30.924 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 aa1700fc-f35b-4ac9-812c-60d0b8a9b874 2024-04-25T09:34:30.924 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 aa1700fc-f35b-4ac9-812c-60d0b8a9b874' 2024-04-25T09:34:30.924 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:34:30.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCmIypmh10jORAAhV4QCeCzfc7Sl3wr/Fy0qQ== 2024-04-25T09:34:30.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCmIypmh10jORAAhV4QCeCzfc7Sl3wr/Fy0qQ=="}' 2024-04-25T09:34:30.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new aa1700fc-f35b-4ac9-812c-60d0b8a9b874 -i td/pg-split-merge/0/new.json 2024-04-25T09:34:31.231 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:34:31.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/pg-split-merge/0/new.json 2024-04-25T09:34:31.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCmIypmh10jORAAhV4QCeCzfc7Sl3wr/Fy0qQ== --osd-uuid aa1700fc-f35b-4ac9-812c-60d0b8a9b874 2024-04-25T09:34:31.287 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:31.299+0000 7f30f46197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:31.299 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:31.311+0000 7f30f46197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:31.300 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:31.315+0000 7f30f46197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:31.301 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:31.315+0000 7f30f46197c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:34:31.301 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:31.315+0000 7f30f46197c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:34:31.301 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:31.315+0000 7f30f46197c0 -1 bluestore(td/pg-split-merge/0/block) _read_bdev_label failed to open td/pg-split-merge/0/block: (2) No such file or directory 2024-04-25T09:34:31.339 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:31.351+0000 7f30f46197c0 -1 bluestore(td/pg-split-merge/0) _read_fsid unparsable uuid 2024-04-25T09:34:37.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/pg-split-merge/0/keyring 2024-04-25T09:34:37.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:34:37.632 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:34:37.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:34:37.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/pg-split-merge/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:34:38.191 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:34:38.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:34:38.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:34:38.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:34:38.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:34:38.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:34:38.224 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:38.239+0000 7f20caa4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:38.248 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:38.263+0000 7f20caa4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:38.249 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:38.263+0000 7f20caa4f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:34:38.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:34:38.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:34:38.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:34:38.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:34:38.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:34:38.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:38.608 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:34:38.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:34:38.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:38.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:38.809 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:38.823+0000 7f20caa4f7c0 -1 Falling back to public interface 2024-04-25T09:34:39.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:40.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:40.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:40.020 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:34:40.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:34:40.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:40.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:40.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:41.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:41.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:41.425 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:34:41.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:34:41.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:41.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:41.428 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:34:41.443+0000 7f20caa4f7c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:34:41.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:42.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:42.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:42.892 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:34:42.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:34:42.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:42.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:43.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:44.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:44.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:44.338 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:34:44.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:34:44.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:44.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:44.770 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:45.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:45.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:45.772 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:34:45.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:34:45.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:45.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:46.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:47.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:47.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:47.335 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:34:47.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:34:47.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:47.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:47.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:48.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:48.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:48.789 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:34:48.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:34:48.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:48.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:49.356 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:50.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:50.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:50.358 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:34:50.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:34:50.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:50.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:50.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:51.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:51.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:51.819 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:34:51.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:34:51.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:51.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:52.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:53.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:53.343 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:53.344 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:34:53.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:34:53.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:53.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:53.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:54.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:54.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:54.813 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:34:54.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:34:54.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:54.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:55.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:56.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:56.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:56.829 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:34:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:34:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:56.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:57.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:34:58.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:34:58.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:34:58.737 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:34:58.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:34:58.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:34:58.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:34:59.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:00.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:00.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:00.587 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:35:00.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:35:00.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:00.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:01.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:02.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:02.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:02.419 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:35:02.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:35:02.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:02.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:03.315 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:04.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:04.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:04.317 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:35:04.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:35:04.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:04.318 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:05.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:06.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:06.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:06.502 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:35:06.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:35:06.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:06.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:06.979 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3044277411,v1:127.0.0.1:6803/3044277411] [v2:127.0.0.1:6804/3044277411,v1:127.0.0.1:6805/3044277411] exists,up aa1700fc-f35b-4ac9-812c-60d0b8a9b874 2024-04-25T09:35:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:35:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:35:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:35:06.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:169: TEST_import_after_split: ceph osd pool create foo 1 2024-04-25T09:35:07.495 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'foo' already exists 2024-04-25T09:35:07.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:170: TEST_import_after_split: wait_for_clean 2024-04-25T09:35:07.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:35:07.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:35:07.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:35:07.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:35:07.512 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:35:07.512 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:35:07.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:35:07.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:35:07.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:35:07.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:35:07.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:35:07.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:35:07.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:35:07.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:35:07.614 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:35:08.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:35:08.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:35:08.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:35:08.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:35:08.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836490 2024-04-25T09:35:08.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836490 2024-04-25T09:35:08.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836490' 2024-04-25T09:35:08.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:35:08.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836490 2024-04-25T09:35:08.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:35:08.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:35:08.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836490 2024-04-25T09:35:08.229 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:35:08.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836490 2024-04-25T09:35:08.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-04-25T09:35:08.231 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836490 2024-04-25T09:35:08.231 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:35:08.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836488 -lt 21474836490 2024-04-25T09:35:08.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:35:09.718 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:35:09.718 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:35:10.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836490 -lt 21474836490 2024-04-25T09:35:10.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:35:10.131 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:35:10.132 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:35:10.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:35:10.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:35:10.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:35:10.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:35:10.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:35:10.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:35:10.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:35:10.663 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:35:11.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:35:11.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:35:11.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:35:11.069 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:35:11.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:35:11.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:35:11.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:35:11.630 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:171: TEST_import_after_split: rados -p foo bench 3 write -b 1024 --no-cleanup 2024-04-25T09:35:11.683 INFO:tasks.workunit.client.0.smithi134.stdout:hints = 1 2024-04-25T09:35:11.683 INFO:tasks.workunit.client.0.smithi134.stdout:Maintaining 16 concurrent writes of 1024 bytes to objects of size 1024 for up to 3 seconds or 0 objects 2024-04-25T09:35:11.683 INFO:tasks.workunit.client.0.smithi134.stdout:Object prefix: benchmark_data_smithi134_475290 2024-04-25T09:35:11.683 INFO:tasks.workunit.client.0.smithi134.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-04-25T09:35:11.683 INFO:tasks.workunit.client.0.smithi134.stdout: 0 0 0 0 0 0 - 0 2024-04-25T09:35:12.683 INFO:tasks.workunit.client.0.smithi134.stdout: 1 16 113 97 0.0947128 0.0947266 0.117002 0.147905 2024-04-25T09:35:13.683 INFO:tasks.workunit.client.0.smithi134.stdout: 2 16 225 209 0.102036 0.109375 0.184514 0.143109 2024-04-25T09:35:14.683 INFO:tasks.workunit.client.0.smithi134.stdout: 3 16 337 321 0.104478 0.109375 0.142241 0.142814 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Total time run: 3.07776 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Total writes made: 337 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Write size: 1024 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Object size: 1024 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Bandwidth (MB/sec): 0.106929 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Bandwidth: 0.00845728 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Max bandwidth (MB/sec): 0.109375 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Min bandwidth (MB/sec): 0.0947266 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Average IOPS: 109 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev IOPS: 8.66025 2024-04-25T09:35:15.684 INFO:tasks.workunit.client.0.smithi134.stdout:Max IOPS: 112 2024-04-25T09:35:15.685 INFO:tasks.workunit.client.0.smithi134.stdout:Min IOPS: 97 2024-04-25T09:35:15.685 INFO:tasks.workunit.client.0.smithi134.stdout:Average Latency(s): 0.142775 2024-04-25T09:35:15.685 INFO:tasks.workunit.client.0.smithi134.stdout:Stddev Latency(s): 0.0237486 2024-04-25T09:35:15.685 INFO:tasks.workunit.client.0.smithi134.stdout:Max latency(s): 0.210273 2024-04-25T09:35:15.685 INFO:tasks.workunit.client.0.smithi134.stdout:Min latency(s): 0.112917 2024-04-25T09:35:15.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:173: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:35:15.758 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:35:15.759 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:35:15.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:35:15.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:35:15.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:35:16.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:35:16.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:174: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op export --pgid 1.0 --file td/pg-split-merge/1.0 --force 2024-04-25T09:35:16.287 INFO:tasks.workunit.client.0.smithi134.stderr:Exporting 1.0 info 1.0( v 9'338 (0'0,9'338] local-lis/les=7/8 n=338 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:35:16.300 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:008edae6:::benchmark_data_smithi134_475290_object316:head# 2024-04-25T09:35:16.301 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:00c23d42:::benchmark_data_smithi134_475290_object117:head# 2024-04-25T09:35:16.301 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:01bd3ff3:::benchmark_data_smithi134_475290_object134:head# 2024-04-25T09:35:16.302 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:027658dc:::benchmark_data_smithi134_475290_object105:head# 2024-04-25T09:35:16.302 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:03856cde:::benchmark_data_smithi134_475290_object102:head# 2024-04-25T09:35:16.303 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:040feddc:::benchmark_data_smithi134_475290_object288:head# 2024-04-25T09:35:16.304 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0432c541:::benchmark_data_smithi134_475290_object245:head# 2024-04-25T09:35:16.305 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:04416736:::benchmark_data_smithi134_475290_object286:head# 2024-04-25T09:35:16.305 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:07489381:::benchmark_data_smithi134_475290_object246:head# 2024-04-25T09:35:16.306 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0b53c6aa:::benchmark_data_smithi134_475290_object290:head# 2024-04-25T09:35:16.306 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0cce342b:::benchmark_data_smithi134_475290_object150:head# 2024-04-25T09:35:16.306 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0cf15861:::benchmark_data_smithi134_475290_object3:head# 2024-04-25T09:35:16.315 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0ece6904:::benchmark_data_smithi134_475290_object149:head# 2024-04-25T09:35:16.316 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0f29c50a:::benchmark_data_smithi134_475290_object97:head# 2024-04-25T09:35:16.317 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:0f53d226:::benchmark_data_smithi134_475290_object20:head# 2024-04-25T09:35:16.318 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:10688876:::benchmark_data_smithi134_475290_object318:head# 2024-04-25T09:35:16.318 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:10a54e36:::benchmark_data_smithi134_475290_object237:head# 2024-04-25T09:35:16.320 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:10d31108:::benchmark_data_smithi134_475290_object153:head# 2024-04-25T09:35:16.321 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1243a9fa:::benchmark_data_smithi134_475290_object212:head# 2024-04-25T09:35:16.321 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:124960ac:::benchmark_data_smithi134_475290_object264:head# 2024-04-25T09:35:16.322 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:12f097f5:::benchmark_data_smithi134_475290_object186:head# 2024-04-25T09:35:16.322 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:136a3336:::benchmark_data_smithi134_475290_object173:head# 2024-04-25T09:35:16.323 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:15523be2:::benchmark_data_smithi134_475290_object50:head# 2024-04-25T09:35:16.323 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:15b7cecb:::benchmark_data_smithi134_475290_object14:head# 2024-04-25T09:35:16.324 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:15d3440c:::benchmark_data_smithi134_475290_object148:head# 2024-04-25T09:35:16.324 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1770c2f5:::benchmark_data_smithi134_475290_object233:head# 2024-04-25T09:35:16.324 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:17ad374b:::benchmark_data_smithi134_475290_object73:head# 2024-04-25T09:35:16.325 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:189ac208:::benchmark_data_smithi134_475290_object230:head# 2024-04-25T09:35:16.326 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:18abdd70:::benchmark_data_smithi134_475290_object329:head# 2024-04-25T09:35:16.326 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:19004c1f:::benchmark_data_smithi134_475290_object99:head# 2024-04-25T09:35:16.326 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:19a59cef:::benchmark_data_smithi134_475290_object158:head# 2024-04-25T09:35:16.327 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1a326068:::benchmark_data_smithi134_475290_object250:head# 2024-04-25T09:35:16.327 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1b4d6ed9:::benchmark_data_smithi134_475290_object223:head# 2024-04-25T09:35:16.328 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1b77b01d:::benchmark_data_smithi134_475290_object17:head# 2024-04-25T09:35:16.329 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1ba1fa94:::benchmark_data_smithi134_475290_object229:head# 2024-04-25T09:35:16.330 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1c69facc:::benchmark_data_smithi134_475290_object89:head# 2024-04-25T09:35:16.330 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1f1c9dc0:::benchmark_data_smithi134_475290_object23:head# 2024-04-25T09:35:16.331 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1fd5c666:::benchmark_data_smithi134_475290_object93:head# 2024-04-25T09:35:16.331 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1fd86c26:::benchmark_data_smithi134_475290_object197:head# 2024-04-25T09:35:16.332 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:1ff3f9d8:::benchmark_data_smithi134_475290_object78:head# 2024-04-25T09:35:16.332 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:21286928:::benchmark_data_smithi134_475290_object179:head# 2024-04-25T09:35:16.333 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:21fba737:::benchmark_data_smithi134_475290_object156:head# 2024-04-25T09:35:16.333 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2253576a:::benchmark_data_smithi134_475290_object314:head# 2024-04-25T09:35:16.334 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2277cfc9:::benchmark_data_smithi134_475290_object181:head# 2024-04-25T09:35:16.334 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:22dc27ae:::benchmark_data_smithi134_475290_object249:head# 2024-04-25T09:35:16.334 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2306d0cf:::benchmark_data_smithi134_475290_object125:head# 2024-04-25T09:35:16.335 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2472a8b4:::benchmark_data_smithi134_475290_object56:head# 2024-04-25T09:35:16.336 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:25100caf:::benchmark_data_smithi134_475290_object322:head# 2024-04-25T09:35:16.336 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:28688137:::benchmark_data_smithi134_475290_object220:head# 2024-04-25T09:35:16.337 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2926fc32:::benchmark_data_smithi134_475290_object319:head# 2024-04-25T09:35:16.337 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:292c0e20:::benchmark_data_smithi134_475290_object70:head# 2024-04-25T09:35:16.338 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2948c0d8:::benchmark_data_smithi134_475290_object64:head# 2024-04-25T09:35:16.338 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:296181c0:::benchmark_data_smithi134_475290_object142:head# 2024-04-25T09:35:16.339 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2a22e72b:::benchmark_data_smithi134_475290_object172:head# 2024-04-25T09:35:16.339 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2c21c3a0:::benchmark_data_smithi134_475290_object59:head# 2024-04-25T09:35:16.341 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2cb5dece:::benchmark_data_smithi134_475290_object46:head# 2024-04-25T09:35:16.341 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2d732dd4:::benchmark_data_smithi134_475290_object235:head# 2024-04-25T09:35:16.342 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2d900852:::benchmark_data_smithi134_475290_object141:head# 2024-04-25T09:35:16.342 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2e298b11:::benchmark_data_smithi134_475290_object330:head# 2024-04-25T09:35:16.343 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2e933e35:::benchmark_data_smithi134_475290_object243:head# 2024-04-25T09:35:16.343 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2f058992:::benchmark_data_smithi134_475290_object266:head# 2024-04-25T09:35:16.344 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:2fd1636e:::benchmark_data_smithi134_475290_object77:head# 2024-04-25T09:35:16.345 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:305cb856:::benchmark_data_smithi134_475290_object0:head# 2024-04-25T09:35:16.345 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:32627073:::benchmark_data_smithi134_475290_object39:head# 2024-04-25T09:35:16.346 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:33ef8bfb:::benchmark_data_smithi134_475290_object333:head# 2024-04-25T09:35:16.346 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:33faf73b:::benchmark_data_smithi134_475290_object155:head# 2024-04-25T09:35:16.347 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:341d142f:::benchmark_data_smithi134_475290_object11:head# 2024-04-25T09:35:16.347 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:358ecffd:::benchmark_data_smithi134_475290_object224:head# 2024-04-25T09:35:16.348 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:375fd66a:::benchmark_data_smithi134_475290_object182:head# 2024-04-25T09:35:16.348 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:37a5b459:::benchmark_data_smithi134_475290_object80:head# 2024-04-25T09:35:16.349 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3851e03a:::benchmark_data_smithi134_475290_object184:head# 2024-04-25T09:35:16.350 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:392b50cc:::benchmark_data_smithi134_475290_object43:head# 2024-04-25T09:35:16.350 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:396eaad0:::benchmark_data_smithi134_475290_object19:head# 2024-04-25T09:35:16.351 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3daa1a19:::benchmark_data_smithi134_475290_object289:head# 2024-04-25T09:35:16.351 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3e3f2862:::benchmark_data_smithi134_475290_object1:head# 2024-04-25T09:35:16.352 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3e6b81f7:::benchmark_data_smithi134_475290_object325:head# 2024-04-25T09:35:16.352 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3e768410:::benchmark_data_smithi134_475290_object272:head# 2024-04-25T09:35:16.353 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3e890388:::benchmark_data_smithi134_475290_object154:head# 2024-04-25T09:35:16.353 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3f510cde:::benchmark_data_smithi134_475290_object211:head# 2024-04-25T09:35:16.354 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3fab6ae6:::benchmark_data_smithi134_475290_object302:head# 2024-04-25T09:35:16.354 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:3fe87bff:::benchmark_data_smithi134_475290_object108:head# 2024-04-25T09:35:16.355 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4254f2d9:::benchmark_data_smithi134_475290_object308:head# 2024-04-25T09:35:16.355 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:429bcd7f:::benchmark_data_smithi134_475290_object115:head# 2024-04-25T09:35:16.356 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:432c7ecf:::benchmark_data_smithi134_475290_object231:head# 2024-04-25T09:35:16.356 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:44265e10:::benchmark_data_smithi134_475290_object256:head# 2024-04-25T09:35:16.356 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4482b722:::benchmark_data_smithi134_475290_object41:head# 2024-04-25T09:35:16.357 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:45ab2314:::benchmark_data_smithi134_475290_object163:head# 2024-04-25T09:35:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:47dd0c11:::benchmark_data_smithi134_475290_object92:head# 2024-04-25T09:35:16.358 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:483cb371:::benchmark_data_smithi134_475290_object236:head# 2024-04-25T09:35:16.359 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4862279b:::benchmark_data_smithi134_475290_object6:head# 2024-04-25T09:35:16.360 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4970920c:::benchmark_data_smithi134_475290_object119:head# 2024-04-25T09:35:16.360 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:49cf7ce2:::benchmark_data_smithi134_475290_object215:head# 2024-04-25T09:35:16.361 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:49e48450:::benchmark_data_smithi134_475290_object255:head# 2024-04-25T09:35:16.361 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4a1804fc:::benchmark_data_smithi134_475290_object112:head# 2024-04-25T09:35:16.362 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4a5d39d4:::benchmark_data_smithi134_475290_object140:head# 2024-04-25T09:35:16.362 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4affbde2:::benchmark_data_smithi134_475290_object79:head# 2024-04-25T09:35:16.363 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4bd1f97d:::benchmark_data_smithi134_475290_object280:head# 2024-04-25T09:35:16.363 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4bfe80fc:::benchmark_data_smithi134_475290_object171:head# 2024-04-25T09:35:16.364 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4dbffa63:::benchmark_data_smithi134_475290_object166:head# 2024-04-25T09:35:16.364 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:4e70418c:::benchmark_data_smithi134_475290_object100:head# 2024-04-25T09:35:16.365 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5009b327:::benchmark_data_smithi134_475290_object9:head# 2024-04-25T09:35:16.365 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:51764d9d:::benchmark_data_smithi134_475290_object53:head# 2024-04-25T09:35:16.366 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:51917f86:::benchmark_data_smithi134_475290_object312:head# 2024-04-25T09:35:16.367 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:53325726:::benchmark_data_smithi134_475290_object18:head# 2024-04-25T09:35:16.367 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5612025a:::benchmark_data_smithi134_475290_object60:head# 2024-04-25T09:35:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:56493bf4:::benchmark_data_smithi134_475290_object152:head# 2024-04-25T09:35:16.368 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:56b77b81:::benchmark_data_smithi134_475290_object267:head# 2024-04-25T09:35:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:56d5ac8d:::benchmark_data_smithi134_475290_object276:head# 2024-04-25T09:35:16.369 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5752fb02:::benchmark_data_smithi134_475290_object48:head# 2024-04-25T09:35:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:59485ee0:::benchmark_data_smithi134_475290_object110:head# 2024-04-25T09:35:16.370 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5b904ba5:::benchmark_data_smithi134_475290_object96:head# 2024-04-25T09:35:16.371 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5d6b020e:::benchmark_data_smithi134_475290_object209:head# 2024-04-25T09:35:16.371 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5dd29384:::benchmark_data_smithi134_475290_object168:head# 2024-04-25T09:35:16.372 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5dd5324c:::benchmark_data_smithi134_475290_object294:head# 2024-04-25T09:35:16.372 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5e5e2030:::benchmark_data_smithi134_475290_object216:head# 2024-04-25T09:35:16.373 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5efbb54a:::benchmark_data_smithi134_475290_object146:head# 2024-04-25T09:35:16.373 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5f4b6e03:::benchmark_data_smithi134_475290_object51:head# 2024-04-25T09:35:16.374 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:5f85e462:::benchmark_data_smithi134_475290_object303:head# 2024-04-25T09:35:16.375 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6070b42c:::benchmark_data_smithi134_475290_object118:head# 2024-04-25T09:35:16.375 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:61230325:::benchmark_data_smithi134_475290_object269:head# 2024-04-25T09:35:16.376 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:628d70d9:::benchmark_data_smithi134_475290_object296:head# 2024-04-25T09:35:16.376 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:634ee02f:::benchmark_data_smithi134_475290_object332:head# 2024-04-25T09:35:16.377 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:63dc5ed6:::benchmark_data_smithi134_475290_object54:head# 2024-04-25T09:35:16.377 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:63e86c4a:::benchmark_data_smithi134_475290_object275:head# 2024-04-25T09:35:16.378 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:64072e10:::benchmark_data_smithi134_475290_object157:head# 2024-04-25T09:35:16.378 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:64f180ad:::benchmark_data_smithi134_475290_object335:head# 2024-04-25T09:35:16.379 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:657345c6:::benchmark_data_smithi134_475290_object83:head# 2024-04-25T09:35:16.380 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:659c3139:::benchmark_data_smithi134_475290_object85:head# 2024-04-25T09:35:16.380 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:66035e87:::benchmark_data_smithi134_475290_object189:head# 2024-04-25T09:35:16.381 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:666b0716:::benchmark_data_smithi134_475290_object101:head# 2024-04-25T09:35:16.381 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:67763e5c:::benchmark_data_smithi134_475290_object24:head# 2024-04-25T09:35:16.382 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:67dc79c1:::benchmark_data_smithi134_475290_object299:head# 2024-04-25T09:35:16.383 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:67e36710:::benchmark_data_smithi134_475290_object240:head# 2024-04-25T09:35:16.384 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:67e5d3d1:::benchmark_data_smithi134_475290_object227:head# 2024-04-25T09:35:16.384 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6851902b:::benchmark_data_smithi134_475290_object151:head# 2024-04-25T09:35:16.385 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:696ac52a:::benchmark_data_smithi134_475290_object297:head# 2024-04-25T09:35:16.385 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6a5c090f:::benchmark_data_smithi134_475290_object22:head# 2024-04-25T09:35:16.386 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6a5c182f:::benchmark_data_smithi134_475290_object320:head# 2024-04-25T09:35:16.386 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6a7c0cee:::benchmark_data_smithi134_475290_object72:head# 2024-04-25T09:35:16.387 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6afec5fa:::benchmark_data_smithi134_475290_object203:head# 2024-04-25T09:35:16.388 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6b1387bc:::benchmark_data_smithi134_475290_object25:head# 2024-04-25T09:35:16.388 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6b4707e6:::benchmark_data_smithi134_475290_object219:head# 2024-04-25T09:35:16.389 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6b50ced5:::benchmark_data_smithi134_475290_object244:head# 2024-04-25T09:35:16.389 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6b612d36:::benchmark_data_smithi134_475290_object265:head# 2024-04-25T09:35:16.390 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6b8cff90:::benchmark_data_smithi134_475290_object273:head# 2024-04-25T09:35:16.390 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6beaad59:::benchmark_data_smithi134_475290_object251:head# 2024-04-25T09:35:16.391 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6cabfcf8:::benchmark_data_smithi134_475290_object114:head# 2024-04-25T09:35:16.391 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6d592614:::benchmark_data_smithi134_475290_object187:head# 2024-04-25T09:35:16.391 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6f782d73:::benchmark_data_smithi134_475290_object133:head# 2024-04-25T09:35:16.392 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:6fec5c79:::benchmark_data_smithi134_475290_object260:head# 2024-04-25T09:35:16.392 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:704a176c:::benchmark_data_smithi134_475290_object121:head# 2024-04-25T09:35:16.393 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:70cd7ab0:::benchmark_data_smithi134_475290_object127:head# 2024-04-25T09:35:16.393 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:73116c32:::benchmark_data_smithi134_475290_object106:head# 2024-04-25T09:35:16.393 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:73a12ab8:::benchmark_data_smithi134_475290_object309:head# 2024-04-25T09:35:16.393 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:75ad16c7:::benchmark_data_smithi134_475290_object221:head# 2024-04-25T09:35:16.394 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7620318f:::benchmark_data_smithi134_475290_object57:head# 2024-04-25T09:35:16.395 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7896bbc6:::benchmark_data_smithi134_475290_object279:head# 2024-04-25T09:35:16.396 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:78d9bc1d:::benchmark_data_smithi134_475290_object32:head# 2024-04-25T09:35:16.396 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:78e0217e:::benchmark_data_smithi134_475290_object188:head# 2024-04-25T09:35:16.397 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:78f8fa70:::benchmark_data_smithi134_475290_object76:head# 2024-04-25T09:35:16.397 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:78fb033a:::benchmark_data_smithi134_475290_object16:head# 2024-04-25T09:35:16.398 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7a64d3be:::benchmark_data_smithi134_475290_object123:head# 2024-04-25T09:35:16.398 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7a891dab:::benchmark_data_smithi134_475290_object40:head# 2024-04-25T09:35:16.399 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7abb9bc6:::benchmark_data_smithi134_475290_object145:head# 2024-04-25T09:35:16.399 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7b885d98:::benchmark_data_smithi134_475290_object29:head# 2024-04-25T09:35:16.400 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7cf7d252:::benchmark_data_smithi134_475290_object81:head# 2024-04-25T09:35:16.401 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7d143424:::benchmark_data_smithi134_475290_object169:head# 2024-04-25T09:35:16.401 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7d2f53f6:::benchmark_data_smithi134_475290_object208:head# 2024-04-25T09:35:16.402 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7dcfbc59:::benchmark_data_smithi134_475290_object135:head# 2024-04-25T09:35:16.402 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7e076b42:::benchmark_data_smithi134_475290_object204:head# 2024-04-25T09:35:16.403 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7f76231c:::benchmark_data_smithi134_475290_object201:head# 2024-04-25T09:35:16.403 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:7f95ab70:::benchmark_data_smithi134_475290_object107:head# 2024-04-25T09:35:16.403 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8309176b:::benchmark_data_smithi134_475290_object98:head# 2024-04-25T09:35:16.404 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:830b40f6:::benchmark_data_smithi134_475290_object317:head# 2024-04-25T09:35:16.404 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:836f1430:::benchmark_data_smithi134_475290_object293:head# 2024-04-25T09:35:16.405 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:84abf74f:::benchmark_data_smithi134_475290_object38:head# 2024-04-25T09:35:16.406 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:86d884d7:::benchmark_data_smithi134_475290_object225:head# 2024-04-25T09:35:16.406 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:87ecec48:::benchmark_data_smithi134_475290_object183:head# 2024-04-25T09:35:16.407 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:884dad08:::benchmark_data_smithi134_475290_object7:head# 2024-04-25T09:35:16.408 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:88abe7b2:::benchmark_data_smithi134_475290_object331:head# 2024-04-25T09:35:16.408 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:88f4648b:::benchmark_data_smithi134_475290_object35:head# 2024-04-25T09:35:16.409 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8a0773b7:::benchmark_data_smithi134_475290_object291:head# 2024-04-25T09:35:16.409 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8a9c61c6:::benchmark_data_smithi134_475290_object281:head# 2024-04-25T09:35:16.410 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8ad05e48:::benchmark_data_smithi134_475290_object239:head# 2024-04-25T09:35:16.410 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8b116db9:::benchmark_data_smithi134_475290_object144:head# 2024-04-25T09:35:16.411 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8bb8d7a6:::benchmark_data_smithi134_475290_object194:head# 2024-04-25T09:35:16.411 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8c65b0e1:::benchmark_data_smithi134_475290_object200:head# 2024-04-25T09:35:16.412 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8c7701f7:::benchmark_data_smithi134_475290_object327:head# 2024-04-25T09:35:16.412 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8c822be8:::benchmark_data_smithi134_475290_object132:head# 2024-04-25T09:35:16.413 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8dd6e43d:::benchmark_data_smithi134_475290_object26:head# 2024-04-25T09:35:16.414 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8e4f09a4:::benchmark_data_smithi134_475290_object178:head# 2024-04-25T09:35:16.414 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8f38ec76:::benchmark_data_smithi134_475290_object285:head# 2024-04-25T09:35:16.415 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:8f72e852:::benchmark_data_smithi134_475290_object131:head# 2024-04-25T09:35:16.416 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:921a6096:::benchmark_data_smithi134_475290_object311:head# 2024-04-25T09:35:16.416 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:927893d1:::benchmark_data_smithi134_475290_object74:head# 2024-04-25T09:35:16.417 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:927ad24a:::benchmark_data_smithi134_475290_object61:head# 2024-04-25T09:35:16.417 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:92c3674c:::benchmark_data_smithi134_475290_object55:head# 2024-04-25T09:35:16.418 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:92f47aa2:::benchmark_data_smithi134_475290_object283:head# 2024-04-25T09:35:16.418 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:93c9ea49:::benchmark_data_smithi134_475290_object190:head# 2024-04-25T09:35:16.419 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:94505c60:::benchmark_data_smithi134_475290_object120:head# 2024-04-25T09:35:16.419 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:94af15fa:::benchmark_data_smithi134_475290_object202:head# 2024-04-25T09:35:16.420 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:95373183:::benchmark_data_smithi134_475290_object323:head# 2024-04-25T09:35:16.420 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:95be0c52:::benchmark_data_smithi134_475290_object68:head# 2024-04-25T09:35:16.421 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:96331eb2:::benchmark_data_smithi134_475290_object44:head# 2024-04-25T09:35:16.421 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9757c613:::benchmark_data_smithi134_475290_object12:head# 2024-04-25T09:35:16.422 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:982bb043:::benchmark_data_smithi134_475290_object305:head# 2024-04-25T09:35:16.422 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:98306725:::benchmark_data_smithi134_475290_object69:head# 2024-04-25T09:35:16.423 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:98b879fe:::benchmark_data_smithi134_475290_object124:head# 2024-04-25T09:35:16.423 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9a3f603d:::benchmark_data_smithi134_475290_object334:head# 2024-04-25T09:35:16.424 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9a9bb596:::benchmark_data_smithi134_475290_object129:head# 2024-04-25T09:35:16.424 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9cb61910:::benchmark_data_smithi134_475290_object111:head# 2024-04-25T09:35:16.425 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9ce2b7e5:::benchmark_data_smithi134_475290_object91:head# 2024-04-25T09:35:16.426 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9d38f049:::benchmark_data_smithi134_475290_object94:head# 2024-04-25T09:35:16.426 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9e68724a:::benchmark_data_smithi134_475290_object175:head# 2024-04-25T09:35:16.427 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9e6ba50e:::benchmark_data_smithi134_475290_object137:head# 2024-04-25T09:35:16.427 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9e86d68e:::benchmark_data_smithi134_475290_object205:head# 2024-04-25T09:35:16.427 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:9e914996:::benchmark_data_smithi134_475290_object10:head# 2024-04-25T09:35:16.428 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a0c0e47c:::benchmark_data_smithi134_475290_object88:head# 2024-04-25T09:35:16.429 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a26e1b17:::benchmark_data_smithi134_475290_object65:head# 2024-04-25T09:35:16.430 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a3a0a90b:::benchmark_data_smithi134_475290_object192:head# 2024-04-25T09:35:16.430 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a3fce679:::benchmark_data_smithi134_475290_object30:head# 2024-04-25T09:35:16.431 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a476671f:::benchmark_data_smithi134_475290_object170:head# 2024-04-25T09:35:16.431 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a61fbf02:::benchmark_data_smithi134_475290_object195:head# 2024-04-25T09:35:16.432 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a63fc976:::benchmark_data_smithi134_475290_object199:head# 2024-04-25T09:35:16.432 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a78b5968:::benchmark_data_smithi134_475290_object2:head# 2024-04-25T09:35:16.433 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a7a7bdf4:::benchmark_data_smithi134_475290_object196:head# 2024-04-25T09:35:16.433 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:a9574739:::benchmark_data_smithi134_475290_object143:head# 2024-04-25T09:35:16.434 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:aa37bbea:::benchmark_data_smithi134_475290_object162:head# 2024-04-25T09:35:16.435 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ab120b47:::benchmark_data_smithi134_475290_object126:head# 2024-04-25T09:35:16.435 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ab824262:::benchmark_data_smithi134_475290_object304:head# 2024-04-25T09:35:16.436 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ac13c284:::benchmark_data_smithi134_475290_object258:head# 2024-04-25T09:35:16.436 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:aca8da7c:::benchmark_data_smithi134_475290_object128:head# 2024-04-25T09:35:16.437 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ad812a99:::benchmark_data_smithi134_475290_object252:head# 2024-04-25T09:35:16.438 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ae375ee8:::benchmark_data_smithi134_475290_object300:head# 2024-04-25T09:35:16.438 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:af5829c5:::benchmark_data_smithi134_475290_object36:head# 2024-04-25T09:35:16.439 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b46a3ae3:::benchmark_data_smithi134_475290_object213:head# 2024-04-25T09:35:16.439 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b74b73bb:::benchmark_data_smithi134_475290_object67:head# 2024-04-25T09:35:16.440 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b7ba4914:::benchmark_data_smithi134_475290_object262:head# 2024-04-25T09:35:16.440 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b8306e8e:::benchmark_data_smithi134_475290_object21:head# 2024-04-25T09:35:16.441 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b8398570:::benchmark_data_smithi134_475290_object328:head# 2024-04-25T09:35:16.441 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:35:16.442 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b95f49dd:::benchmark_data_smithi134_475290_object180:head# 2024-04-25T09:35:16.443 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b97d9eee:::benchmark_data_smithi134_475290_object241:head# 2024-04-25T09:35:16.443 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:b9c9e84f:::benchmark_data_smithi134_475290_object37:head# 2024-04-25T09:35:16.444 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ba141183:::benchmark_data_smithi134_475290_object45:head# 2024-04-25T09:35:16.444 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bc22d806:::benchmark_data_smithi134_475290_object313:head# 2024-04-25T09:35:16.444 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bd558d39:::benchmark_data_smithi134_475290_object47:head# 2024-04-25T09:35:16.445 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bdc2b7cb:::benchmark_data_smithi134_475290_object218:head# 2024-04-25T09:35:16.446 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:be103679:::benchmark_data_smithi134_475290_object87:head# 2024-04-25T09:35:16.446 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:be41fc66:::benchmark_data_smithi134_475290_object193:head# 2024-04-25T09:35:16.447 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:be86854c:::benchmark_data_smithi134_475290_object167:head# 2024-04-25T09:35:16.448 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:beb9051e:::benchmark_data_smithi134_475290_object242:head# 2024-04-25T09:35:16.448 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:bfa87029:::benchmark_data_smithi134_475290_object248:head# 2024-04-25T09:35:16.449 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c07d9d94:::benchmark_data_smithi134_475290_object228:head# 2024-04-25T09:35:16.449 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c26db96a:::benchmark_data_smithi134_475290_object160:head# 2024-04-25T09:35:16.449 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c2dc89e0:::benchmark_data_smithi134_475290_object116:head# 2024-04-25T09:35:16.450 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c2f0e4ad:::benchmark_data_smithi134_475290_object165:head# 2024-04-25T09:35:16.450 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c37dffc3:::benchmark_data_smithi134_475290_object310:head# 2024-04-25T09:35:16.451 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c5909792:::benchmark_data_smithi134_475290_object28:head# 2024-04-25T09:35:16.452 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c5e821ce:::benchmark_data_smithi134_475290_object52:head# 2024-04-25T09:35:16.452 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c6a7ece0:::benchmark_data_smithi134_475290_object159:head# 2024-04-25T09:35:16.453 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c71874cb:::benchmark_data_smithi134_475290_object253:head# 2024-04-25T09:35:16.453 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c744ff84:::benchmark_data_smithi134_475290_object257:head# 2024-04-25T09:35:16.454 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c778b670:::benchmark_data_smithi134_475290_object139:head# 2024-04-25T09:35:16.454 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c7d40f20:::benchmark_data_smithi134_475290_object238:head# 2024-04-25T09:35:16.455 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c8c0f822:::benchmark_data_smithi134_475290_object284:head# 2024-04-25T09:35:16.455 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:c925f11f:::benchmark_data_smithi134_475290_object254:head# 2024-04-25T09:35:16.455 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ca84e5b0:::benchmark_data_smithi134_475290_object292:head# 2024-04-25T09:35:16.456 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cc986898:::benchmark_data_smithi134_475290_object214:head# 2024-04-25T09:35:16.456 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cdf4ca4e:::benchmark_data_smithi134_475290_object75:head# 2024-04-25T09:35:16.457 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ce4303ac:::benchmark_data_smithi134_475290_object206:head# 2024-04-25T09:35:16.458 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cf2e00c6:::benchmark_data_smithi134_475290_object49:head# 2024-04-25T09:35:16.458 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cf787146:::benchmark_data_smithi134_475290_object282:head# 2024-04-25T09:35:16.459 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cf87cccb:::benchmark_data_smithi134_475290_object336:head# 2024-04-25T09:35:16.459 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cfe87a57:::benchmark_data_smithi134_475290_object210:head# 2024-04-25T09:35:16.460 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:cff56803:::benchmark_data_smithi134_475290_object63:head# 2024-04-25T09:35:16.460 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d0fdfde4:::benchmark_data_smithi134_475290_object164:head# 2024-04-25T09:35:16.461 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d12c4d05:::benchmark_data_smithi134_475290_object185:head# 2024-04-25T09:35:16.461 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d363dbf8:::benchmark_data_smithi134_475290_object268:head# 2024-04-25T09:35:16.462 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d4d952c9:::benchmark_data_smithi134_475290_object247:head# 2024-04-25T09:35:16.462 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d596705d:::benchmark_data_smithi134_475290_object86:head# 2024-04-25T09:35:16.463 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d5cbc83f:::benchmark_data_smithi134_475290_object130:head# 2024-04-25T09:35:16.463 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d7ba6bc9:::benchmark_data_smithi134_475290_object234:head# 2024-04-25T09:35:16.464 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:d9c2bc02:::benchmark_data_smithi134_475290_object277:head# 2024-04-25T09:35:16.464 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:da1e92d1:::benchmark_data_smithi134_475290_object191:head# 2024-04-25T09:35:16.464 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dc51f732:::benchmark_data_smithi134_475290_object207:head# 2024-04-25T09:35:16.465 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dc73b204:::benchmark_data_smithi134_475290_object324:head# 2024-04-25T09:35:16.465 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd1840ba:::benchmark_data_smithi134_475290_object5:head# 2024-04-25T09:35:16.466 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd2471c2:::benchmark_data_smithi134_475290_object176:head# 2024-04-25T09:35:16.467 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:dd41b4ba:::benchmark_data_smithi134_475290_object103:head# 2024-04-25T09:35:16.467 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:de4d823a:::benchmark_data_smithi134_475290_object15:head# 2024-04-25T09:35:16.468 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:de801fb8:::benchmark_data_smithi134_475290_object321:head# 2024-04-25T09:35:16.468 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:df377bfa:::benchmark_data_smithi134_475290_object104:head# 2024-04-25T09:35:16.469 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e145fa5c:::benchmark_data_smithi134_475290_object287:head# 2024-04-25T09:35:16.469 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e2060fc5:::benchmark_data_smithi134_475290_object42:head# 2024-04-25T09:35:16.470 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e2b8bbb7:::benchmark_data_smithi134_475290_object27:head# 2024-04-25T09:35:16.470 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e3ed32ad:::benchmark_data_smithi134_475290_object8:head# 2024-04-25T09:35:16.471 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e407da68:::benchmark_data_smithi134_475290_object177:head# 2024-04-25T09:35:16.471 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e4533166:::benchmark_data_smithi134_475290_object315:head# 2024-04-25T09:35:16.472 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e5034cee:::benchmark_data_smithi134_475290_object13:head# 2024-04-25T09:35:16.472 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e5229009:::benchmark_data_smithi134_475290_object58:head# 2024-04-25T09:35:16.473 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e5dfcb2e:::benchmark_data_smithi134_475290_object90:head# 2024-04-25T09:35:16.474 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e63fc978:::benchmark_data_smithi134_475290_object113:head# 2024-04-25T09:35:16.474 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e647b6f8:::benchmark_data_smithi134_475290_object147:head# 2024-04-25T09:35:16.475 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e763e731:::benchmark_data_smithi134_475290_object226:head# 2024-04-25T09:35:16.475 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e7f5edc1:::benchmark_data_smithi134_475290_object270:head# 2024-04-25T09:35:16.476 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e98eea6b:::benchmark_data_smithi134_475290_object326:head# 2024-04-25T09:35:16.476 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:e9b3053a:::benchmark_data_smithi134_475290_object4:head# 2024-04-25T09:35:16.477 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ea30affa:::benchmark_data_smithi134_475290_object232:head# 2024-04-25T09:35:16.478 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ea4fa371:::benchmark_data_smithi134_475290_object259:head# 2024-04-25T09:35:16.478 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ea7499be:::benchmark_data_smithi134_475290_object295:head# 2024-04-25T09:35:16.479 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ea8da7aa:::benchmark_data_smithi134_475290_object263:head# 2024-04-25T09:35:16.479 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ebc6b0bd:::benchmark_data_smithi134_475290_object298:head# 2024-04-25T09:35:16.480 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ec4bf37f:::benchmark_data_smithi134_475290_object122:head# 2024-04-25T09:35:16.480 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ec4d7fd5:::benchmark_data_smithi134_475290_object34:head# 2024-04-25T09:35:16.481 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ed51e633:::benchmark_data_smithi134_475290_object33:head# 2024-04-25T09:35:16.482 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:edccdca8:::benchmark_data_smithi134_475290_object66:head# 2024-04-25T09:35:16.482 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ee2e615e:::benchmark_data_smithi134_475290_object198:head# 2024-04-25T09:35:16.483 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:eff184b6:::benchmark_data_smithi134_475290_object174:head# 2024-04-25T09:35:16.483 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f0f4f861:::benchmark_data_smithi134_475290_object222:head# 2024-04-25T09:35:16.484 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f149f50f:::benchmark_data_smithi134_475290_object274:head# 2024-04-25T09:35:16.484 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f1772bf4:::benchmark_data_smithi134_475290_object82:head# 2024-04-25T09:35:16.485 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f188431f:::benchmark_data_smithi134_475290_object95:head# 2024-04-25T09:35:16.485 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f3889df6:::benchmark_data_smithi134_475290_object109:head# 2024-04-25T09:35:16.486 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f470a9d0:::benchmark_data_smithi134_475290_object261:head# 2024-04-25T09:35:16.486 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f4cff160:::benchmark_data_smithi134_475290_object84:head# 2024-04-25T09:35:16.487 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f4f1c0f0:::benchmark_data_smithi134_475290_object138:head# 2024-04-25T09:35:16.487 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f4ff7503:::benchmark_data_smithi134_475290_object306:head# 2024-04-25T09:35:16.488 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f56edaec:::benchmark_data_smithi134_475290_object217:head# 2024-04-25T09:35:16.488 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f5df4d30:::benchmark_data_smithi134_475290_object161:head# 2024-04-25T09:35:16.488 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f6cc03ef:::benchmark_data_smithi134_475290_object62:head# 2024-04-25T09:35:16.490 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:f8303239:::benchmark_data_smithi134_475290_object307:head# 2024-04-25T09:35:16.490 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fc20dc47:::benchmark_data_smithi134_475290_object301:head# 2024-04-25T09:35:16.491 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fcce1c39:::benchmark_data_smithi134_475290_object278:head# 2024-04-25T09:35:16.491 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:fe9210d7:::benchmark_data_smithi134_475290_object136:head# 2024-04-25T09:35:16.492 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ff7863e6:::benchmark_data_smithi134_475290_object271:head# 2024-04-25T09:35:16.493 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ff9c75cb:::benchmark_data_smithi134_475290_object31:head# 2024-04-25T09:35:16.493 INFO:tasks.workunit.client.0.smithi134.stderr:Read #1:ffd2cc7f:::benchmark_data_smithi134_475290_object71:head# 2024-04-25T09:35:16.494 INFO:tasks.workunit.client.0.smithi134.stderr:Export successful 2024-04-25T09:35:16.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:175: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:35:16.786 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:35:16.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:35:16.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:35:16.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:35:16.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:35:16.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:35:16.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:35:16.787 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:35:16.788 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:35:16.788 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:16.789 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:35:16.789 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:35:16.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:35:16.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:35:16.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:35:16.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:35:16.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:35:16.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:35:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:35:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:35:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:35:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:35:16.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:35:16.791 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:35:16.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:35:16.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:35:16.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:35:16.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:35:16.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:35:16.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:35:16.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:35:16.824 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:16.838+0000 7fc4ea2d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:16.827 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:16.838+0000 7fc4ea2d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:16.829 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:16.842+0000 7fc4ea2d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:17.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:35:17.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:35:17.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:35:17.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:35:17.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:35:17.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:17.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:35:17.205 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:35:17.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:17.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:17.373 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:17.386+0000 7fc4ea2d87c0 -1 Falling back to public interface 2024-04-25T09:35:17.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:18.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:18.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:18.655 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:35:18.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:35:18.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:18.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:19.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:20.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:20.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:20.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:35:20.081 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:35:20.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:20.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:20.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:20.713 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:20.726+0000 7fc4ea2d87c0 -1 osd.0 9 log_to_monitors true 2024-04-25T09:35:21.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:21.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:21.524 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:35:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:35:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:21.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:22.040 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 12 up_thru 12 down_at 10 last_clean_interval [5,9) [v2:127.0.0.1:6802/2045822032,v1:127.0.0.1:6803/2045822032] [v2:127.0.0.1:6804/2045822032,v1:127.0.0.1:6805/2045822032] exists,up aa1700fc-f35b-4ac9-812c-60d0b8a9b874 2024-04-25T09:35:22.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:35:22.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:35:22.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:35:22.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:177: TEST_import_after_split: ceph osd pool set foo pg_num 2 2024-04-25T09:35:22.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:178: TEST_import_after_split: sleep 5 2024-04-25T09:35:27.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: ceph daemon osd.0 perf dump 2024-04-25T09:35:27.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: jq .osd.numpg 2024-04-25T09:35:27.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:179: TEST_import_after_split: grep 1 2024-04-25T09:35:27.883 INFO:tasks.workunit.client.0.smithi134.stderr:Can't get admin socket path: unable to get conf option admin_socket for osd: b"error parsing 'osd': expected string of the form TYPE.ID, valid types are: auth, mon, osd, mds, mgr, client\n" 2024-04-25T09:35:27.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:180: TEST_import_after_split: wait_for_clean 2024-04-25T09:35:27.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:35:27.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:35:27.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:35:27.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:35:27.887 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:35:27.887 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:35:27.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:35:27.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:35:27.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:35:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:35:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:35:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:35:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:35:27.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:35:27.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:35:28.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:35:28.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:35:28.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:35:28.413 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:35:28.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=51539607556 2024-04-25T09:35:28.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 51539607556 2024-04-25T09:35:28.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-51539607556' 2024-04-25T09:35:28.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:35:28.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-51539607556 2024-04-25T09:35:28.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:35:28.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:35:28.582 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-51539607556 2024-04-25T09:35:28.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:35:28.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=51539607556 2024-04-25T09:35:28.585 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 51539607556 2024-04-25T09:35:28.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 51539607556' 2024-04-25T09:35:28.585 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:35:28.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607554 -lt 51539607556 2024-04-25T09:35:28.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:35:29.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:35:29.986 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:35:30.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 51539607556 -lt 51539607556 2024-04-25T09:35:30.405 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:35:30.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:35:30.406 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:35:30.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:35:30.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:35:30.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:35:30.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:35:30.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:35:30.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:35:30.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:35:30.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:35:31.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:35:31.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:35:31.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:35:31.376 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:35:31.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:35:31.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:35:31.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:35:31.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:182: TEST_import_after_split: kill_daemons td/pg-split-merge TERM osd.0 2024-04-25T09:35:31.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:35:31.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:35:31.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:35:31.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:35:31.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:35:32.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:35:32.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:184: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.0 --force 2024-04-25T09:35:33.569 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T09:35:33.569 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T09:35:33.569 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.0_head removing 1.0 2024-04-25T09:35:33.944 INFO:tasks.workunit.client.0.smithi134.stdout:Remove successful 2024-04-25T09:35:34.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:187: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:35:35.318 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-04-25T09:35:35.318 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.0 2024-04-25T09:35:35.319 INFO:tasks.workunit.client.0.smithi134.stdout:pool 1 pg_num 1 -> 2 2024-04-25T09:35:35.319 INFO:tasks.workunit.client.0.smithi134.stderr: children are 1.1 2024-04-25T09:35:35.319 INFO:tasks.workunit.client.0.smithi134.stderr:pgid 1.0 splits to 1.1 and 1.1 exists 2024-04-25T09:35:35.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:189: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op remove --pgid 1.1 --force 2024-04-25T09:35:36.528 INFO:tasks.workunit.client.0.smithi134.stdout: marking collection for removal 2024-04-25T09:35:36.528 INFO:tasks.workunit.client.0.smithi134.stdout:setting '_remove' omap key 2024-04-25T09:35:36.528 INFO:tasks.workunit.client.0.smithi134.stdout:finish_remove_pgs 1.1_head removing 1.1 2024-04-25T09:35:36.859 INFO:tasks.workunit.client.0.smithi134.stdout:Remove successful 2024-04-25T09:35:37.314 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:191: TEST_import_after_split: ceph-objectstore-tool --data-path td/pg-split-merge/0 --op import --pgid 1.0 --file td/pg-split-merge/1.0 2024-04-25T09:35:38.290 INFO:tasks.workunit.client.0.smithi134.stdout:get_pg_num_history pg_num_history pg_num_history(e18 pg_nums {1={7=1,16=2}} deleted_pools ) 2024-04-25T09:35:38.290 INFO:tasks.workunit.client.0.smithi134.stdout:Importing pgid 1.0 2024-04-25T09:35:38.291 INFO:tasks.workunit.client.0.smithi134.stdout:pool 1 pg_num 1 -> 2 2024-04-25T09:35:38.291 INFO:tasks.workunit.client.0.smithi134.stderr: children are 1.1 2024-04-25T09:35:38.291 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:008edae6:::benchmark_data_smithi134_475290_object316:head# 2024-04-25T09:35:38.291 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.379 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:00c23d42:::benchmark_data_smithi134_475290_object117:head# 2024-04-25T09:35:38.379 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.429 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:01bd3ff3:::benchmark_data_smithi134_475290_object134:head# 2024-04-25T09:35:38.429 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.479 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:027658dc:::benchmark_data_smithi134_475290_object105:head# 2024-04-25T09:35:38.479 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.529 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:03856cde:::benchmark_data_smithi134_475290_object102:head# 2024-04-25T09:35:38.530 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.580 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:040feddc:::benchmark_data_smithi134_475290_object288:head# 2024-04-25T09:35:38.580 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.630 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0432c541:::benchmark_data_smithi134_475290_object245:head# 2024-04-25T09:35:38.630 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.672 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:04416736:::benchmark_data_smithi134_475290_object286:head# 2024-04-25T09:35:38.672 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.713 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:07489381:::benchmark_data_smithi134_475290_object246:head# 2024-04-25T09:35:38.713 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.755 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0b53c6aa:::benchmark_data_smithi134_475290_object290:head# 2024-04-25T09:35:38.755 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.797 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0cce342b:::benchmark_data_smithi134_475290_object150:head# 2024-04-25T09:35:38.797 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.839 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0cf15861:::benchmark_data_smithi134_475290_object3:head# 2024-04-25T09:35:38.839 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.880 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0ece6904:::benchmark_data_smithi134_475290_object149:head# 2024-04-25T09:35:38.881 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.922 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0f29c50a:::benchmark_data_smithi134_475290_object97:head# 2024-04-25T09:35:38.922 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:38.964 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:0f53d226:::benchmark_data_smithi134_475290_object20:head# 2024-04-25T09:35:38.964 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.006 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:10688876:::benchmark_data_smithi134_475290_object318:head# 2024-04-25T09:35:39.006 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.048 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:10a54e36:::benchmark_data_smithi134_475290_object237:head# 2024-04-25T09:35:39.048 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.089 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:10d31108:::benchmark_data_smithi134_475290_object153:head# 2024-04-25T09:35:39.090 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.131 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1243a9fa:::benchmark_data_smithi134_475290_object212:head# 2024-04-25T09:35:39.131 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.173 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:124960ac:::benchmark_data_smithi134_475290_object264:head# 2024-04-25T09:35:39.173 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.206 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:12f097f5:::benchmark_data_smithi134_475290_object186:head# 2024-04-25T09:35:39.208 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.240 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:136a3336:::benchmark_data_smithi134_475290_object173:head# 2024-04-25T09:35:39.240 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.282 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:15523be2:::benchmark_data_smithi134_475290_object50:head# 2024-04-25T09:35:39.282 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.323 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:15b7cecb:::benchmark_data_smithi134_475290_object14:head# 2024-04-25T09:35:39.324 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.365 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:15d3440c:::benchmark_data_smithi134_475290_object148:head# 2024-04-25T09:35:39.365 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.407 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1770c2f5:::benchmark_data_smithi134_475290_object233:head# 2024-04-25T09:35:39.407 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.441 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:17ad374b:::benchmark_data_smithi134_475290_object73:head# 2024-04-25T09:35:39.441 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.482 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:189ac208:::benchmark_data_smithi134_475290_object230:head# 2024-04-25T09:35:39.482 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.566 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:18abdd70:::benchmark_data_smithi134_475290_object329:head# 2024-04-25T09:35:39.566 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.625 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:19004c1f:::benchmark_data_smithi134_475290_object99:head# 2024-04-25T09:35:39.625 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.666 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:19a59cef:::benchmark_data_smithi134_475290_object158:head# 2024-04-25T09:35:39.666 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.708 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1a326068:::benchmark_data_smithi134_475290_object250:head# 2024-04-25T09:35:39.708 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.742 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1b4d6ed9:::benchmark_data_smithi134_475290_object223:head# 2024-04-25T09:35:39.742 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.784 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1b77b01d:::benchmark_data_smithi134_475290_object17:head# 2024-04-25T09:35:39.784 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.825 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1ba1fa94:::benchmark_data_smithi134_475290_object229:head# 2024-04-25T09:35:39.825 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.867 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1c69facc:::benchmark_data_smithi134_475290_object89:head# 2024-04-25T09:35:39.867 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.909 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1f1c9dc0:::benchmark_data_smithi134_475290_object23:head# 2024-04-25T09:35:39.909 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:39.951 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1fd5c666:::benchmark_data_smithi134_475290_object93:head# 2024-04-25T09:35:39.951 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.003 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1fd86c26:::benchmark_data_smithi134_475290_object197:head# 2024-04-25T09:35:40.003 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.093 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:1ff3f9d8:::benchmark_data_smithi134_475290_object78:head# 2024-04-25T09:35:40.094 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.177 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:21286928:::benchmark_data_smithi134_475290_object179:head# 2024-04-25T09:35:40.177 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.210 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:21fba737:::benchmark_data_smithi134_475290_object156:head# 2024-04-25T09:35:40.210 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.244 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2253576a:::benchmark_data_smithi134_475290_object314:head# 2024-04-25T09:35:40.244 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.277 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2277cfc9:::benchmark_data_smithi134_475290_object181:head# 2024-04-25T09:35:40.277 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.310 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:22dc27ae:::benchmark_data_smithi134_475290_object249:head# 2024-04-25T09:35:40.310 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.344 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2306d0cf:::benchmark_data_smithi134_475290_object125:head# 2024-04-25T09:35:40.344 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.377 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2472a8b4:::benchmark_data_smithi134_475290_object56:head# 2024-04-25T09:35:40.378 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.411 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:25100caf:::benchmark_data_smithi134_475290_object322:head# 2024-04-25T09:35:40.411 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.445 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:28688137:::benchmark_data_smithi134_475290_object220:head# 2024-04-25T09:35:40.445 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.478 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2926fc32:::benchmark_data_smithi134_475290_object319:head# 2024-04-25T09:35:40.478 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.511 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:292c0e20:::benchmark_data_smithi134_475290_object70:head# 2024-04-25T09:35:40.511 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.545 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2948c0d8:::benchmark_data_smithi134_475290_object64:head# 2024-04-25T09:35:40.545 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.578 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:296181c0:::benchmark_data_smithi134_475290_object142:head# 2024-04-25T09:35:40.578 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.612 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2a22e72b:::benchmark_data_smithi134_475290_object172:head# 2024-04-25T09:35:40.612 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.645 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2c21c3a0:::benchmark_data_smithi134_475290_object59:head# 2024-04-25T09:35:40.645 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.679 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2cb5dece:::benchmark_data_smithi134_475290_object46:head# 2024-04-25T09:35:40.679 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.712 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2d732dd4:::benchmark_data_smithi134_475290_object235:head# 2024-04-25T09:35:40.712 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.746 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2d900852:::benchmark_data_smithi134_475290_object141:head# 2024-04-25T09:35:40.746 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.779 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2e298b11:::benchmark_data_smithi134_475290_object330:head# 2024-04-25T09:35:40.779 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.813 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2e933e35:::benchmark_data_smithi134_475290_object243:head# 2024-04-25T09:35:40.813 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.854 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2f058992:::benchmark_data_smithi134_475290_object266:head# 2024-04-25T09:35:40.855 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.888 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:2fd1636e:::benchmark_data_smithi134_475290_object77:head# 2024-04-25T09:35:40.888 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.921 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:305cb856:::benchmark_data_smithi134_475290_object0:head# 2024-04-25T09:35:40.921 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:40.963 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:32627073:::benchmark_data_smithi134_475290_object39:head# 2024-04-25T09:35:40.963 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.005 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:33ef8bfb:::benchmark_data_smithi134_475290_object333:head# 2024-04-25T09:35:41.005 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.055 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:33faf73b:::benchmark_data_smithi134_475290_object155:head# 2024-04-25T09:35:41.055 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.105 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:341d142f:::benchmark_data_smithi134_475290_object11:head# 2024-04-25T09:35:41.105 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.156 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:358ecffd:::benchmark_data_smithi134_475290_object224:head# 2024-04-25T09:35:41.156 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.230 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:375fd66a:::benchmark_data_smithi134_475290_object182:head# 2024-04-25T09:35:41.230 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.272 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:37a5b459:::benchmark_data_smithi134_475290_object80:head# 2024-04-25T09:35:41.272 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.322 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3851e03a:::benchmark_data_smithi134_475290_object184:head# 2024-04-25T09:35:41.323 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.364 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:392b50cc:::benchmark_data_smithi134_475290_object43:head# 2024-04-25T09:35:41.364 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.406 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:396eaad0:::benchmark_data_smithi134_475290_object19:head# 2024-04-25T09:35:41.406 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.448 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3daa1a19:::benchmark_data_smithi134_475290_object289:head# 2024-04-25T09:35:41.448 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.490 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3e3f2862:::benchmark_data_smithi134_475290_object1:head# 2024-04-25T09:35:41.490 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.565 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3e6b81f7:::benchmark_data_smithi134_475290_object325:head# 2024-04-25T09:35:41.565 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.615 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3e768410:::benchmark_data_smithi134_475290_object272:head# 2024-04-25T09:35:41.615 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.657 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3e890388:::benchmark_data_smithi134_475290_object154:head# 2024-04-25T09:35:41.657 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.699 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3f510cde:::benchmark_data_smithi134_475290_object211:head# 2024-04-25T09:35:41.699 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.740 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3fab6ae6:::benchmark_data_smithi134_475290_object302:head# 2024-04-25T09:35:41.740 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.782 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:3fe87bff:::benchmark_data_smithi134_475290_object108:head# 2024-04-25T09:35:41.782 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.824 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4254f2d9:::benchmark_data_smithi134_475290_object308:head# 2024-04-25T09:35:41.824 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.866 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:429bcd7f:::benchmark_data_smithi134_475290_object115:head# 2024-04-25T09:35:41.866 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.907 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:432c7ecf:::benchmark_data_smithi134_475290_object231:head# 2024-04-25T09:35:41.907 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.949 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:44265e10:::benchmark_data_smithi134_475290_object256:head# 2024-04-25T09:35:41.949 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:41.991 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4482b722:::benchmark_data_smithi134_475290_object41:head# 2024-04-25T09:35:41.991 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.085 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:45ab2314:::benchmark_data_smithi134_475290_object163:head# 2024-04-25T09:35:42.086 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.161 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:47dd0c11:::benchmark_data_smithi134_475290_object92:head# 2024-04-25T09:35:42.161 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.219 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:483cb371:::benchmark_data_smithi134_475290_object236:head# 2024-04-25T09:35:42.219 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.261 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4862279b:::benchmark_data_smithi134_475290_object6:head# 2024-04-25T09:35:42.261 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.303 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4970920c:::benchmark_data_smithi134_475290_object119:head# 2024-04-25T09:35:42.303 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.345 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:49cf7ce2:::benchmark_data_smithi134_475290_object215:head# 2024-04-25T09:35:42.345 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.386 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:49e48450:::benchmark_data_smithi134_475290_object255:head# 2024-04-25T09:35:42.386 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.428 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4a1804fc:::benchmark_data_smithi134_475290_object112:head# 2024-04-25T09:35:42.428 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.470 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4a5d39d4:::benchmark_data_smithi134_475290_object140:head# 2024-04-25T09:35:42.470 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.512 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4affbde2:::benchmark_data_smithi134_475290_object79:head# 2024-04-25T09:35:42.512 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.554 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4bd1f97d:::benchmark_data_smithi134_475290_object280:head# 2024-04-25T09:35:42.554 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.595 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4bfe80fc:::benchmark_data_smithi134_475290_object171:head# 2024-04-25T09:35:42.595 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.637 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4dbffa63:::benchmark_data_smithi134_475290_object166:head# 2024-04-25T09:35:42.637 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.683 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:4e70418c:::benchmark_data_smithi134_475290_object100:head# 2024-04-25T09:35:42.683 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.721 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5009b327:::benchmark_data_smithi134_475290_object9:head# 2024-04-25T09:35:42.721 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.762 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:51764d9d:::benchmark_data_smithi134_475290_object53:head# 2024-04-25T09:35:42.763 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.804 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:51917f86:::benchmark_data_smithi134_475290_object312:head# 2024-04-25T09:35:42.804 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.846 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:53325726:::benchmark_data_smithi134_475290_object18:head# 2024-04-25T09:35:42.846 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.888 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5612025a:::benchmark_data_smithi134_475290_object60:head# 2024-04-25T09:35:42.888 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.921 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:56493bf4:::benchmark_data_smithi134_475290_object152:head# 2024-04-25T09:35:42.921 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.954 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:56b77b81:::benchmark_data_smithi134_475290_object267:head# 2024-04-25T09:35:42.955 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:42.988 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:56d5ac8d:::benchmark_data_smithi134_475290_object276:head# 2024-04-25T09:35:42.988 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.021 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5752fb02:::benchmark_data_smithi134_475290_object48:head# 2024-04-25T09:35:43.022 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.063 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:59485ee0:::benchmark_data_smithi134_475290_object110:head# 2024-04-25T09:35:43.063 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.105 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5b904ba5:::benchmark_data_smithi134_475290_object96:head# 2024-04-25T09:35:43.105 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.147 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5d6b020e:::benchmark_data_smithi134_475290_object209:head# 2024-04-25T09:35:43.147 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.180 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5dd29384:::benchmark_data_smithi134_475290_object168:head# 2024-04-25T09:35:43.180 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.214 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5dd5324c:::benchmark_data_smithi134_475290_object294:head# 2024-04-25T09:35:43.214 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.247 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5e5e2030:::benchmark_data_smithi134_475290_object216:head# 2024-04-25T09:35:43.248 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.281 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5efbb54a:::benchmark_data_smithi134_475290_object146:head# 2024-04-25T09:35:43.281 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.322 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5f4b6e03:::benchmark_data_smithi134_475290_object51:head# 2024-04-25T09:35:43.323 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.364 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:5f85e462:::benchmark_data_smithi134_475290_object303:head# 2024-04-25T09:35:43.364 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.414 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6070b42c:::benchmark_data_smithi134_475290_object118:head# 2024-04-25T09:35:43.415 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.464 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:61230325:::benchmark_data_smithi134_475290_object269:head# 2024-04-25T09:35:43.465 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.515 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:628d70d9:::benchmark_data_smithi134_475290_object296:head# 2024-04-25T09:35:43.515 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.565 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:634ee02f:::benchmark_data_smithi134_475290_object332:head# 2024-04-25T09:35:43.565 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.607 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:63dc5ed6:::benchmark_data_smithi134_475290_object54:head# 2024-04-25T09:35:43.607 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.649 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:63e86c4a:::benchmark_data_smithi134_475290_object275:head# 2024-04-25T09:35:43.649 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.690 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:64072e10:::benchmark_data_smithi134_475290_object157:head# 2024-04-25T09:35:43.691 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.732 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:64f180ad:::benchmark_data_smithi134_475290_object335:head# 2024-04-25T09:35:43.732 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.774 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:657345c6:::benchmark_data_smithi134_475290_object83:head# 2024-04-25T09:35:43.774 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.816 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:659c3139:::benchmark_data_smithi134_475290_object85:head# 2024-04-25T09:35:43.816 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.857 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:66035e87:::benchmark_data_smithi134_475290_object189:head# 2024-04-25T09:35:43.858 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.899 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:666b0716:::benchmark_data_smithi134_475290_object101:head# 2024-04-25T09:35:43.899 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.941 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67763e5c:::benchmark_data_smithi134_475290_object24:head# 2024-04-25T09:35:43.941 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:43.983 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67dc79c1:::benchmark_data_smithi134_475290_object299:head# 2024-04-25T09:35:43.983 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.058 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67e36710:::benchmark_data_smithi134_475290_object240:head# 2024-04-25T09:35:44.058 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.126 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:67e5d3d1:::benchmark_data_smithi134_475290_object227:head# 2024-04-25T09:35:44.127 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.201 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6851902b:::benchmark_data_smithi134_475290_object151:head# 2024-04-25T09:35:44.201 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.242 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:696ac52a:::benchmark_data_smithi134_475290_object297:head# 2024-04-25T09:35:44.243 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.284 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6a5c090f:::benchmark_data_smithi134_475290_object22:head# 2024-04-25T09:35:44.285 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.326 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6a5c182f:::benchmark_data_smithi134_475290_object320:head# 2024-04-25T09:35:44.326 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.368 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6a7c0cee:::benchmark_data_smithi134_475290_object72:head# 2024-04-25T09:35:44.368 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.410 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6afec5fa:::benchmark_data_smithi134_475290_object203:head# 2024-04-25T09:35:44.410 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.451 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6b1387bc:::benchmark_data_smithi134_475290_object25:head# 2024-04-25T09:35:44.452 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.493 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6b4707e6:::benchmark_data_smithi134_475290_object219:head# 2024-04-25T09:35:44.493 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.535 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6b50ced5:::benchmark_data_smithi134_475290_object244:head# 2024-04-25T09:35:44.535 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.577 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6b612d36:::benchmark_data_smithi134_475290_object265:head# 2024-04-25T09:35:44.577 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.619 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6b8cff90:::benchmark_data_smithi134_475290_object273:head# 2024-04-25T09:35:44.619 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.661 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6beaad59:::benchmark_data_smithi134_475290_object251:head# 2024-04-25T09:35:44.661 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.702 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6cabfcf8:::benchmark_data_smithi134_475290_object114:head# 2024-04-25T09:35:44.703 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.744 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6d592614:::benchmark_data_smithi134_475290_object187:head# 2024-04-25T09:35:44.745 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.794 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6f782d73:::benchmark_data_smithi134_475290_object133:head# 2024-04-25T09:35:44.794 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.844 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:6fec5c79:::benchmark_data_smithi134_475290_object260:head# 2024-04-25T09:35:44.845 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.886 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:704a176c:::benchmark_data_smithi134_475290_object121:head# 2024-04-25T09:35:44.886 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.930 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:70cd7ab0:::benchmark_data_smithi134_475290_object127:head# 2024-04-25T09:35:44.931 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:44.970 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:73116c32:::benchmark_data_smithi134_475290_object106:head# 2024-04-25T09:35:44.970 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.011 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:73a12ab8:::benchmark_data_smithi134_475290_object309:head# 2024-04-25T09:35:45.012 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.078 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:75ad16c7:::benchmark_data_smithi134_475290_object221:head# 2024-04-25T09:35:45.079 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.120 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7620318f:::benchmark_data_smithi134_475290_object57:head# 2024-04-25T09:35:45.120 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.162 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7896bbc6:::benchmark_data_smithi134_475290_object279:head# 2024-04-25T09:35:45.162 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.204 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:78d9bc1d:::benchmark_data_smithi134_475290_object32:head# 2024-04-25T09:35:45.204 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.245 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:78e0217e:::benchmark_data_smithi134_475290_object188:head# 2024-04-25T09:35:45.245 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.287 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:78f8fa70:::benchmark_data_smithi134_475290_object76:head# 2024-04-25T09:35:45.287 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.329 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:78fb033a:::benchmark_data_smithi134_475290_object16:head# 2024-04-25T09:35:45.329 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.379 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7a64d3be:::benchmark_data_smithi134_475290_object123:head# 2024-04-25T09:35:45.379 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.421 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7a891dab:::benchmark_data_smithi134_475290_object40:head# 2024-04-25T09:35:45.421 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.463 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7abb9bc6:::benchmark_data_smithi134_475290_object145:head# 2024-04-25T09:35:45.463 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.511 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7b885d98:::benchmark_data_smithi134_475290_object29:head# 2024-04-25T09:35:45.511 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.588 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7cf7d252:::benchmark_data_smithi134_475290_object81:head# 2024-04-25T09:35:45.589 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.630 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7d143424:::benchmark_data_smithi134_475290_object169:head# 2024-04-25T09:35:45.631 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.672 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7d2f53f6:::benchmark_data_smithi134_475290_object208:head# 2024-04-25T09:35:45.672 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.714 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7dcfbc59:::benchmark_data_smithi134_475290_object135:head# 2024-04-25T09:35:45.714 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.755 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7e076b42:::benchmark_data_smithi134_475290_object204:head# 2024-04-25T09:35:45.756 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.856 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7f76231c:::benchmark_data_smithi134_475290_object201:head# 2024-04-25T09:35:45.857 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.897 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:7f95ab70:::benchmark_data_smithi134_475290_object107:head# 2024-04-25T09:35:45.897 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.939 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8309176b:::benchmark_data_smithi134_475290_object98:head# 2024-04-25T09:35:45.939 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:45.981 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:830b40f6:::benchmark_data_smithi134_475290_object317:head# 2024-04-25T09:35:45.981 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.056 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:836f1430:::benchmark_data_smithi134_475290_object293:head# 2024-04-25T09:35:46.056 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.115 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:84abf74f:::benchmark_data_smithi134_475290_object38:head# 2024-04-25T09:35:46.115 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.182 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:86d884d7:::benchmark_data_smithi134_475290_object225:head# 2024-04-25T09:35:46.182 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.215 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:87ecec48:::benchmark_data_smithi134_475290_object183:head# 2024-04-25T09:35:46.215 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.257 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:884dad08:::benchmark_data_smithi134_475290_object7:head# 2024-04-25T09:35:46.258 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.299 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:88abe7b2:::benchmark_data_smithi134_475290_object331:head# 2024-04-25T09:35:46.299 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.341 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:88f4648b:::benchmark_data_smithi134_475290_object35:head# 2024-04-25T09:35:46.341 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.383 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8a0773b7:::benchmark_data_smithi134_475290_object291:head# 2024-04-25T09:35:46.383 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.424 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8a9c61c6:::benchmark_data_smithi134_475290_object281:head# 2024-04-25T09:35:46.424 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.466 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8ad05e48:::benchmark_data_smithi134_475290_object239:head# 2024-04-25T09:35:46.466 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.508 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8b116db9:::benchmark_data_smithi134_475290_object144:head# 2024-04-25T09:35:46.508 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.550 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8bb8d7a6:::benchmark_data_smithi134_475290_object194:head# 2024-04-25T09:35:46.550 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.591 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c65b0e1:::benchmark_data_smithi134_475290_object200:head# 2024-04-25T09:35:46.592 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.633 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c7701f7:::benchmark_data_smithi134_475290_object327:head# 2024-04-25T09:35:46.633 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.675 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8c822be8:::benchmark_data_smithi134_475290_object132:head# 2024-04-25T09:35:46.675 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.717 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8dd6e43d:::benchmark_data_smithi134_475290_object26:head# 2024-04-25T09:35:46.717 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.759 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8e4f09a4:::benchmark_data_smithi134_475290_object178:head# 2024-04-25T09:35:46.759 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.800 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8f38ec76:::benchmark_data_smithi134_475290_object285:head# 2024-04-25T09:35:46.801 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.842 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:8f72e852:::benchmark_data_smithi134_475290_object131:head# 2024-04-25T09:35:46.842 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.884 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:921a6096:::benchmark_data_smithi134_475290_object311:head# 2024-04-25T09:35:46.884 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.926 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:927893d1:::benchmark_data_smithi134_475290_object74:head# 2024-04-25T09:35:46.926 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:46.968 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:927ad24a:::benchmark_data_smithi134_475290_object61:head# 2024-04-25T09:35:46.968 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.009 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:92c3674c:::benchmark_data_smithi134_475290_object55:head# 2024-04-25T09:35:47.009 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.051 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:92f47aa2:::benchmark_data_smithi134_475290_object283:head# 2024-04-25T09:35:47.051 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.093 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:93c9ea49:::benchmark_data_smithi134_475290_object190:head# 2024-04-25T09:35:47.093 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.135 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:94505c60:::benchmark_data_smithi134_475290_object120:head# 2024-04-25T09:35:47.135 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.176 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:94af15fa:::benchmark_data_smithi134_475290_object202:head# 2024-04-25T09:35:47.176 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.218 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:95373183:::benchmark_data_smithi134_475290_object323:head# 2024-04-25T09:35:47.218 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.260 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:95be0c52:::benchmark_data_smithi134_475290_object68:head# 2024-04-25T09:35:47.260 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.302 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:96331eb2:::benchmark_data_smithi134_475290_object44:head# 2024-04-25T09:35:47.302 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.344 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9757c613:::benchmark_data_smithi134_475290_object12:head# 2024-04-25T09:35:47.344 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.386 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:982bb043:::benchmark_data_smithi134_475290_object305:head# 2024-04-25T09:35:47.386 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.427 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:98306725:::benchmark_data_smithi134_475290_object69:head# 2024-04-25T09:35:47.427 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.469 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:98b879fe:::benchmark_data_smithi134_475290_object124:head# 2024-04-25T09:35:47.469 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.519 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9a3f603d:::benchmark_data_smithi134_475290_object334:head# 2024-04-25T09:35:47.519 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.586 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9a9bb596:::benchmark_data_smithi134_475290_object129:head# 2024-04-25T09:35:47.587 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.645 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9cb61910:::benchmark_data_smithi134_475290_object111:head# 2024-04-25T09:35:47.645 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.695 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9ce2b7e5:::benchmark_data_smithi134_475290_object91:head# 2024-04-25T09:35:47.695 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.745 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9d38f049:::benchmark_data_smithi134_475290_object94:head# 2024-04-25T09:35:47.745 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.787 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e68724a:::benchmark_data_smithi134_475290_object175:head# 2024-04-25T09:35:47.787 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.837 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e6ba50e:::benchmark_data_smithi134_475290_object137:head# 2024-04-25T09:35:47.837 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.879 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e86d68e:::benchmark_data_smithi134_475290_object205:head# 2024-04-25T09:35:47.879 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.920 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:9e914996:::benchmark_data_smithi134_475290_object10:head# 2024-04-25T09:35:47.921 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:47.971 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a0c0e47c:::benchmark_data_smithi134_475290_object88:head# 2024-04-25T09:35:47.971 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.019 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a26e1b17:::benchmark_data_smithi134_475290_object65:head# 2024-04-25T09:35:48.019 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.108 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a3a0a90b:::benchmark_data_smithi134_475290_object192:head# 2024-04-25T09:35:48.109 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.188 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a3fce679:::benchmark_data_smithi134_475290_object30:head# 2024-04-25T09:35:48.188 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.230 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a476671f:::benchmark_data_smithi134_475290_object170:head# 2024-04-25T09:35:48.230 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.272 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a61fbf02:::benchmark_data_smithi134_475290_object195:head# 2024-04-25T09:35:48.272 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.313 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a63fc976:::benchmark_data_smithi134_475290_object199:head# 2024-04-25T09:35:48.314 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.356 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a78b5968:::benchmark_data_smithi134_475290_object2:head# 2024-04-25T09:35:48.356 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.397 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a7a7bdf4:::benchmark_data_smithi134_475290_object196:head# 2024-04-25T09:35:48.397 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.439 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:a9574739:::benchmark_data_smithi134_475290_object143:head# 2024-04-25T09:35:48.439 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.481 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:aa37bbea:::benchmark_data_smithi134_475290_object162:head# 2024-04-25T09:35:48.481 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.523 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ab120b47:::benchmark_data_smithi134_475290_object126:head# 2024-04-25T09:35:48.523 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.564 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ab824262:::benchmark_data_smithi134_475290_object304:head# 2024-04-25T09:35:48.565 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.606 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ac13c284:::benchmark_data_smithi134_475290_object258:head# 2024-04-25T09:35:48.607 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.648 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:aca8da7c:::benchmark_data_smithi134_475290_object128:head# 2024-04-25T09:35:48.648 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.690 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ad812a99:::benchmark_data_smithi134_475290_object252:head# 2024-04-25T09:35:48.690 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.732 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ae375ee8:::benchmark_data_smithi134_475290_object300:head# 2024-04-25T09:35:48.732 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.773 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:af5829c5:::benchmark_data_smithi134_475290_object36:head# 2024-04-25T09:35:48.773 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.815 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b46a3ae3:::benchmark_data_smithi134_475290_object213:head# 2024-04-25T09:35:48.815 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.857 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b74b73bb:::benchmark_data_smithi134_475290_object67:head# 2024-04-25T09:35:48.857 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.932 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b7ba4914:::benchmark_data_smithi134_475290_object262:head# 2024-04-25T09:35:48.932 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:48.974 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b8306e8e:::benchmark_data_smithi134_475290_object21:head# 2024-04-25T09:35:48.974 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.016 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b8398570:::benchmark_data_smithi134_475290_object328:head# 2024-04-25T09:35:49.016 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.049 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b8615e71:::benchmark_last_metadata:head# 2024-04-25T09:35:49.049 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.082 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b95f49dd:::benchmark_data_smithi134_475290_object180:head# 2024-04-25T09:35:49.083 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.116 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b97d9eee:::benchmark_data_smithi134_475290_object241:head# 2024-04-25T09:35:49.116 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.149 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:b9c9e84f:::benchmark_data_smithi134_475290_object37:head# 2024-04-25T09:35:49.149 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.183 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ba141183:::benchmark_data_smithi134_475290_object45:head# 2024-04-25T09:35:49.183 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.216 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bc22d806:::benchmark_data_smithi134_475290_object313:head# 2024-04-25T09:35:49.217 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.250 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bd558d39:::benchmark_data_smithi134_475290_object47:head# 2024-04-25T09:35:49.250 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.283 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bdc2b7cb:::benchmark_data_smithi134_475290_object218:head# 2024-04-25T09:35:49.283 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.317 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:be103679:::benchmark_data_smithi134_475290_object87:head# 2024-04-25T09:35:49.317 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.350 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:be41fc66:::benchmark_data_smithi134_475290_object193:head# 2024-04-25T09:35:49.350 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.392 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:be86854c:::benchmark_data_smithi134_475290_object167:head# 2024-04-25T09:35:49.392 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.433 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:beb9051e:::benchmark_data_smithi134_475290_object242:head# 2024-04-25T09:35:49.434 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.475 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:bfa87029:::benchmark_data_smithi134_475290_object248:head# 2024-04-25T09:35:49.476 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.567 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c07d9d94:::benchmark_data_smithi134_475290_object228:head# 2024-04-25T09:35:49.568 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.617 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c26db96a:::benchmark_data_smithi134_475290_object160:head# 2024-04-25T09:35:49.618 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.659 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c2dc89e0:::benchmark_data_smithi134_475290_object116:head# 2024-04-25T09:35:49.659 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.701 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c2f0e4ad:::benchmark_data_smithi134_475290_object165:head# 2024-04-25T09:35:49.701 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.777 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c37dffc3:::benchmark_data_smithi134_475290_object310:head# 2024-04-25T09:35:49.777 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.818 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c5909792:::benchmark_data_smithi134_475290_object28:head# 2024-04-25T09:35:49.819 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.860 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c5e821ce:::benchmark_data_smithi134_475290_object52:head# 2024-04-25T09:35:49.861 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.902 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c6a7ece0:::benchmark_data_smithi134_475290_object159:head# 2024-04-25T09:35:49.902 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.944 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c71874cb:::benchmark_data_smithi134_475290_object253:head# 2024-04-25T09:35:49.944 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:49.986 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c744ff84:::benchmark_data_smithi134_475290_object257:head# 2024-04-25T09:35:49.986 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.061 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c778b670:::benchmark_data_smithi134_475290_object139:head# 2024-04-25T09:35:50.061 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.120 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c7d40f20:::benchmark_data_smithi134_475290_object238:head# 2024-04-25T09:35:50.120 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.178 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c8c0f822:::benchmark_data_smithi134_475290_object284:head# 2024-04-25T09:35:50.179 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.220 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:c925f11f:::benchmark_data_smithi134_475290_object254:head# 2024-04-25T09:35:50.220 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.262 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ca84e5b0:::benchmark_data_smithi134_475290_object292:head# 2024-04-25T09:35:50.262 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.304 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cc986898:::benchmark_data_smithi134_475290_object214:head# 2024-04-25T09:35:50.304 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.345 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cdf4ca4e:::benchmark_data_smithi134_475290_object75:head# 2024-04-25T09:35:50.346 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.387 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ce4303ac:::benchmark_data_smithi134_475290_object206:head# 2024-04-25T09:35:50.387 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.429 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf2e00c6:::benchmark_data_smithi134_475290_object49:head# 2024-04-25T09:35:50.429 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.471 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf787146:::benchmark_data_smithi134_475290_object282:head# 2024-04-25T09:35:50.471 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.513 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cf87cccb:::benchmark_data_smithi134_475290_object336:head# 2024-04-25T09:35:50.513 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.569 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cfe87a57:::benchmark_data_smithi134_475290_object210:head# 2024-04-25T09:35:50.593 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.671 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:cff56803:::benchmark_data_smithi134_475290_object63:head# 2024-04-25T09:35:50.671 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.755 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d0fdfde4:::benchmark_data_smithi134_475290_object164:head# 2024-04-25T09:35:50.755 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.831 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d12c4d05:::benchmark_data_smithi134_475290_object185:head# 2024-04-25T09:35:50.832 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.931 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d363dbf8:::benchmark_data_smithi134_475290_object268:head# 2024-04-25T09:35:50.931 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:50.990 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d4d952c9:::benchmark_data_smithi134_475290_object247:head# 2024-04-25T09:35:50.990 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.032 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d596705d:::benchmark_data_smithi134_475290_object86:head# 2024-04-25T09:35:51.032 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.073 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d5cbc83f:::benchmark_data_smithi134_475290_object130:head# 2024-04-25T09:35:51.074 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.115 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d7ba6bc9:::benchmark_data_smithi134_475290_object234:head# 2024-04-25T09:35:51.115 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.157 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:d9c2bc02:::benchmark_data_smithi134_475290_object277:head# 2024-04-25T09:35:51.157 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.199 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:da1e92d1:::benchmark_data_smithi134_475290_object191:head# 2024-04-25T09:35:51.199 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.240 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dc51f732:::benchmark_data_smithi134_475290_object207:head# 2024-04-25T09:35:51.241 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.282 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dc73b204:::benchmark_data_smithi134_475290_object324:head# 2024-04-25T09:35:51.282 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.324 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd1840ba:::benchmark_data_smithi134_475290_object5:head# 2024-04-25T09:35:51.324 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.366 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd2471c2:::benchmark_data_smithi134_475290_object176:head# 2024-04-25T09:35:51.366 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.408 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:dd41b4ba:::benchmark_data_smithi134_475290_object103:head# 2024-04-25T09:35:51.408 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.449 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:de4d823a:::benchmark_data_smithi134_475290_object15:head# 2024-04-25T09:35:51.449 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.491 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:de801fb8:::benchmark_data_smithi134_475290_object321:head# 2024-04-25T09:35:51.491 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.583 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:df377bfa:::benchmark_data_smithi134_475290_object104:head# 2024-04-25T09:35:51.584 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.633 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e145fa5c:::benchmark_data_smithi134_475290_object287:head# 2024-04-25T09:35:51.634 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.667 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e2060fc5:::benchmark_data_smithi134_475290_object42:head# 2024-04-25T09:35:51.667 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.700 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e2b8bbb7:::benchmark_data_smithi134_475290_object27:head# 2024-04-25T09:35:51.701 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.734 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e3ed32ad:::benchmark_data_smithi134_475290_object8:head# 2024-04-25T09:35:51.734 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.767 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e407da68:::benchmark_data_smithi134_475290_object177:head# 2024-04-25T09:35:51.767 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.801 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e4533166:::benchmark_data_smithi134_475290_object315:head# 2024-04-25T09:35:51.801 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.834 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e5034cee:::benchmark_data_smithi134_475290_object13:head# 2024-04-25T09:35:51.834 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.868 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e5229009:::benchmark_data_smithi134_475290_object58:head# 2024-04-25T09:35:51.868 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.901 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e5dfcb2e:::benchmark_data_smithi134_475290_object90:head# 2024-04-25T09:35:51.901 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.943 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e63fc978:::benchmark_data_smithi134_475290_object113:head# 2024-04-25T09:35:51.943 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:51.993 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e647b6f8:::benchmark_data_smithi134_475290_object147:head# 2024-04-25T09:35:51.994 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.060 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e763e731:::benchmark_data_smithi134_475290_object226:head# 2024-04-25T09:35:52.061 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.119 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e7f5edc1:::benchmark_data_smithi134_475290_object270:head# 2024-04-25T09:35:52.119 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.169 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e98eea6b:::benchmark_data_smithi134_475290_object326:head# 2024-04-25T09:35:52.169 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.211 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:e9b3053a:::benchmark_data_smithi134_475290_object4:head# 2024-04-25T09:35:52.211 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.253 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ea30affa:::benchmark_data_smithi134_475290_object232:head# 2024-04-25T09:35:52.253 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.295 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ea4fa371:::benchmark_data_smithi134_475290_object259:head# 2024-04-25T09:35:52.295 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.336 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ea7499be:::benchmark_data_smithi134_475290_object295:head# 2024-04-25T09:35:52.336 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.378 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ea8da7aa:::benchmark_data_smithi134_475290_object263:head# 2024-04-25T09:35:52.378 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.420 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ebc6b0bd:::benchmark_data_smithi134_475290_object298:head# 2024-04-25T09:35:52.420 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.462 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ec4bf37f:::benchmark_data_smithi134_475290_object122:head# 2024-04-25T09:35:52.462 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.504 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ec4d7fd5:::benchmark_data_smithi134_475290_object34:head# 2024-04-25T09:35:52.504 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.545 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ed51e633:::benchmark_data_smithi134_475290_object33:head# 2024-04-25T09:35:52.546 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.587 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:edccdca8:::benchmark_data_smithi134_475290_object66:head# 2024-04-25T09:35:52.587 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.629 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ee2e615e:::benchmark_data_smithi134_475290_object198:head# 2024-04-25T09:35:52.629 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.671 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:eff184b6:::benchmark_data_smithi134_475290_object174:head# 2024-04-25T09:35:52.671 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.738 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f0f4f861:::benchmark_data_smithi134_475290_object222:head# 2024-04-25T09:35:52.739 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.779 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f149f50f:::benchmark_data_smithi134_475290_object274:head# 2024-04-25T09:35:52.779 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.855 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f1772bf4:::benchmark_data_smithi134_475290_object82:head# 2024-04-25T09:35:52.855 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.896 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f188431f:::benchmark_data_smithi134_475290_object95:head# 2024-04-25T09:35:52.896 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.938 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f3889df6:::benchmark_data_smithi134_475290_object109:head# 2024-04-25T09:35:52.938 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:52.997 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f470a9d0:::benchmark_data_smithi134_475290_object261:head# 2024-04-25T09:35:52.997 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.038 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f4cff160:::benchmark_data_smithi134_475290_object84:head# 2024-04-25T09:35:53.038 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.097 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f4f1c0f0:::benchmark_data_smithi134_475290_object138:head# 2024-04-25T09:35:53.097 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.139 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f4ff7503:::benchmark_data_smithi134_475290_object306:head# 2024-04-25T09:35:53.139 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.180 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f56edaec:::benchmark_data_smithi134_475290_object217:head# 2024-04-25T09:35:53.181 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.222 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f5df4d30:::benchmark_data_smithi134_475290_object161:head# 2024-04-25T09:35:53.222 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.264 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f6cc03ef:::benchmark_data_smithi134_475290_object62:head# 2024-04-25T09:35:53.264 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.306 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:f8303239:::benchmark_data_smithi134_475290_object307:head# 2024-04-25T09:35:53.306 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.347 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fc20dc47:::benchmark_data_smithi134_475290_object301:head# 2024-04-25T09:35:53.348 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.389 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fcce1c39:::benchmark_data_smithi134_475290_object278:head# 2024-04-25T09:35:53.389 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.431 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:fe9210d7:::benchmark_data_smithi134_475290_object136:head# 2024-04-25T09:35:53.431 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.473 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ff7863e6:::benchmark_data_smithi134_475290_object271:head# 2024-04-25T09:35:53.473 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.526 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ff9c75cb:::benchmark_data_smithi134_475290_object31:head# 2024-04-25T09:35:53.526 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.598 INFO:tasks.workunit.client.0.smithi134.stdout:Write #1:ffd2cc7f:::benchmark_data_smithi134_475290_object71:head# 2024-04-25T09:35:53.598 INFO:tasks.workunit.client.0.smithi134.stdout:snapset 0=[]:{} 2024-04-25T09:35:53.657 INFO:tasks.workunit.client.0.smithi134.stdout:write_pg epoch 9 info 1.0( v 9'338 (0'0,9'338] local-lis/les=7/8 n=338 ec=7/7 lis/c=7/7 les/c/f=8/8/0 sis=7) 2024-04-25T09:35:53.707 INFO:tasks.workunit.client.0.smithi134.stdout:Import successful 2024-04-25T09:35:54.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:193: TEST_import_after_split: activate_osd td/pg-split-merge 0 2024-04-25T09:35:54.246 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/pg-split-merge 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/pg-split-merge/0 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10' 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/pg-split-merge/0' 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/pg-split-merge/0/journal' 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:35:54.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/pg-split-merge' 2024-04-25T09:35:54.248 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:35:54.248 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:35:54.248 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:35:54.248 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:35:54.248 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:35:54.248 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:35:54.249 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:35:54.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' 2024-04-25T09:35:54.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/pg-split-merge/$name.log' 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/pg-split-merge/$name.pid' 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/pg-split-merge/0 2024-04-25T09:35:54.250 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:35:54.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-04-25T09:35:54.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=674facff-f888-41a1-99f8-b1480373e477 --auth-supported=none --mon-host=127.0.0.1:7147 --mon_min_osdmap_epochs=50 --paxos_service_trim_min=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/pg-split-merge/0 --osd-journal=td/pg-split-merge/0/journal --chdir= --run-dir=td/pg-split-merge '--admin-socket=/tmp/ceph-asok.451985/$cluster-$name.asok' --debug-osd=20 '--log-file=td/pg-split-merge/$name.log' '--pid-file=td/pg-split-merge/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:35:54.251 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/pg-split-merge/0/whoami 2024-04-25T09:35:54.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-04-25T09:35:54.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:35:54.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:35:54.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:35:54.285 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:54.301+0000 7f8a7f15d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:54.287 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:54.301+0000 7f8a7f15d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:54.289 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:54.305+0000 7f8a7f15d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:35:54.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-04-25T09:35:54.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:35:54.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:35:54.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:35:54.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:35:54.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:54.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:35:54.685 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:35:54.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:54.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:54.850 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:54.865+0000 7f8a7f15d7c0 -1 Falling back to public interface 2024-04-25T09:35:55.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:56.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:56.173 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:35:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:35:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:56.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:57.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:57.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:57.580 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:35:57.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:35:57.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:57.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:57.908 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:35:57.921+0000 7f8a7f15d7c0 -1 osd.0 18 log_to_monitors true 2024-04-25T09:35:57.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:35:59.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:35:59.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:35:59.001 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:35:59.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:35:59.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:35:59.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:35:59.428 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 21 up_thru 21 down_at 19 last_clean_interval [12,18) [v2:127.0.0.1:6802/90986688,v1:127.0.0.1:6803/90986688] [v2:127.0.0.1:6804/90986688,v1:127.0.0.1:6805/90986688] exists,up aa1700fc-f35b-4ac9-812c-60d0b8a9b874 2024-04-25T09:35:59.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:35:59.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:35:59.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:35:59.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:195: TEST_import_after_split: wait_for_clean 2024-04-25T09:35:59.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:35:59.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:35:59.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:35:59.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:35:59.431 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:35:59.431 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:35:59.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:35:59.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:35:59.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:35:59.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:35:59.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:35:59.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:35:59.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:35:59.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:35:59.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:35:59.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids=0 2024-04-25T09:35:59.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:35:59.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:35:59.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:36:00.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=90194313219 2024-04-25T09:36:00.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 90194313219 2024-04-25T09:36:00.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-90194313219' 2024-04-25T09:36:00.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:36:00.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-90194313219 2024-04-25T09:36:00.124 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:36:00.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:36:00.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-90194313219 2024-04-25T09:36:00.125 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:36:00.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=90194313219 2024-04-25T09:36:00.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 90194313219' 2024-04-25T09:36:00.127 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 90194313219 2024-04-25T09:36:00.127 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:36:00.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 1 -lt 90194313219 2024-04-25T09:36:00.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:36:01.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:36:01.522 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:36:01.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 90194313219 -lt 90194313219 2024-04-25T09:36:01.944 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:36:01.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:36:01.945 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:36:02.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 2 == 0 2024-04-25T09:36:02.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:36:02.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:36:02.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:36:02.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:36:02.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:36:02.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:36:02.492 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:36:02.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=2 2024-04-25T09:36:02.935 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:36:02.935 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:36:02.936 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 2 = 2 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/pg-split-merge.sh:17: run: teardown td/pg-split-merge 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:36:03.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:36:03.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:36:03.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:36:03.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:36:03.593 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:36:03.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:36:03.594 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:36:03.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:36:03.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:36:03.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:36:03.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:36:03.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:36:03.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:36:03.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:36:03.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:36:03.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:36:03.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:36:03.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:36:03.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:03.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:36:03.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451985 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/pg-split-merge 0 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/pg-split-merge 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/pg-split-merge KILL 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:36:03.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:36:03.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:36:03.611 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:36:03.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:36:03.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:36:03.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:36:03.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:36:03.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:36:03.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:36:03.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:36:03.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:36:03.615 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:36:03.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:36:03.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:36:03.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/pg-split-merge 2024-04-25T09:36:03.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:36:03.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:03.617 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.451985 2024-04-25T09:36:03.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.451985 2024-04-25T09:36:03.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:36:03.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:36:03.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:36:03.620 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:36:03.620 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:36:03.669 INFO:tasks.workunit:Running workunit osd/repeer-on-acting-back.sh... 2024-04-25T09:36:03.670 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/repeer-on-acting-back.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh 2024-04-25T09:36:03.724 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:36:03.730 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:36:03.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:36:03.730 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/repeer-on-acting-back 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:22: run: local dir=td/repeer-on-acting-back 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:23: run: shift 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: export poolname=test 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:25: run: poolname=test 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: export testobjects=100 2024-04-25T09:36:03.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:26: run: testobjects=100 2024-04-25T09:36:03.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: export loglen=12 2024-04-25T09:36:03.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:27: run: loglen=12 2024-04-25T09:36:03.732 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: expr 12 / 2 2024-04-25T09:36:03.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: export trim=6 2024-04-25T09:36:03.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:28: run: trim=6 2024-04-25T09:36:03.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: export CEPH_MON=127.0.0.1:7115 2024-04-25T09:36:03.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:29: run: CEPH_MON=127.0.0.1:7115 2024-04-25T09:36:03.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:30: run: export CEPH_ARGS 2024-04-25T09:36:03.732 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: uuidgen 2024-04-25T09:36:03.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:31: run: CEPH_ARGS+='--fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none ' 2024-04-25T09:36:03.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:32: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2024-04-25T09:36:03.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:34: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2024-04-25T09:36:03.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:36: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:36:03.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: set 2024-04-25T09:36:03.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:38: run: local funcs=TEST_repeer_on_down_acting_member_coming_back 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:39: run: for func in $funcs 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:40: run: setup td/repeer-on-acting-back 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repeer-on-acting-back 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repeer-on-acting-back 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:36:03.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:36:03.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:36:03.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:36:03.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:36:03.737 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:36:03.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:36:03.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:36:03.739 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:36:03.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:36:03.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:36:03.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:36:03.741 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:36:03.741 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:36:03.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:36:03.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:36:03.743 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:36:03.743 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:03.743 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:36:03.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.478728 2024-04-25T09:36:03.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:36:03.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:36:03.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repeer-on-acting-back 2024-04-25T09:36:03.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:36:03.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:03.746 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:36:03.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.478728 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repeer-on-acting-back 1' TERM HUP INT 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:41: run: TEST_repeer_on_down_acting_member_coming_back td/repeer-on-acting-back 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:48: TEST_repeer_on_down_acting_member_coming_back: local dir=td/repeer-on-acting-back 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:49: TEST_repeer_on_down_acting_member_coming_back: local dummyfile=/etc/fstab 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:51: TEST_repeer_on_down_acting_member_coming_back: local num_osds=6 2024-04-25T09:36:03.748 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: expr 6 - 1 2024-04-25T09:36:03.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: seq 0 5 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:52: TEST_repeer_on_down_acting_member_coming_back: local 'osds=0 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:53: TEST_repeer_on_down_acting_member_coming_back: run_mon td/repeer-on-acting-back a 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repeer-on-acting-back 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repeer-on-acting-back/a 2024-04-25T09:36:03.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repeer-on-acting-back/a --run-dir=td/repeer-on-acting-back 2024-04-25T09:36:04.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:36:04.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:36:04.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:36:04.234 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:36:04.234 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:04.234 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:36:04.234 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:36:04.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repeer-on-acting-back/a '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --mon-cluster-log-file=td/repeer-on-acting-back/log --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:36:04.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:36:04.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:36:04.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:36:04.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:36:04.452 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:36:04.453 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:36:04.453 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:36:04.453 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:36:04.453 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:36:04.454 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:36:04.454 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:04.454 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:36:04.454 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.478728/ceph-mon.a.asok 2024-04-25T09:36:04.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:36:04.454 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.478728/ceph-mon.a.asok config get fsid 2024-04-25T09:36:04.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:36:04.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:36:04.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:36:04.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.478728/ceph-mon.a.asok 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:36:04.570 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.478728/ceph-mon.a.asok config get mon_host 2024-04-25T09:36:04.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:54: TEST_repeer_on_down_acting_member_coming_back: run_mgr td/repeer-on-acting-back x 2024-04-25T09:36:04.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repeer-on-acting-back 2024-04-25T09:36:04.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:36:04.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:36:04.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:36:04.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repeer-on-acting-back/x 2024-04-25T09:36:04.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:36:05.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:36:05.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:36:05.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:36:05.600 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:36:05.600 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:05.600 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:36:05.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:36:05.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:36:05.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repeer-on-acting-back/x '--log-file=td/repeer-on-acting-back/$name.log' '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --run-dir=td/repeer-on-acting-back '--pid-file=td/repeer-on-acting-back/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 0 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/0 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:36:05.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/0' 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/0/journal' 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:36:05.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/0 2024-04-25T09:36:05.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:36:05.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=824909d2-e965-4842-a8da-9ff7d5dc55a2 2024-04-25T09:36:05.644 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 824909d2-e965-4842-a8da-9ff7d5dc55a2 2024-04-25T09:36:05.644 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 824909d2-e965-4842-a8da-9ff7d5dc55a2' 2024-04-25T09:36:05.644 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:36:05.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAFJCpmG4l7KBAAA0ZGgYXfFODf55O1E3Vihg== 2024-04-25T09:36:05.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAFJCpmG4l7KBAAA0ZGgYXfFODf55O1E3Vihg=="}' 2024-04-25T09:36:05.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 824909d2-e965-4842-a8da-9ff7d5dc55a2 -i td/repeer-on-acting-back/0/new.json 2024-04-25T09:36:05.956 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:36:05.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/0/new.json 2024-04-25T09:36:05.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFJCpmG4l7KBAAA0ZGgYXfFODf55O1E3Vihg== --osd-uuid 824909d2-e965-4842-a8da-9ff7d5dc55a2 2024-04-25T09:36:06.014 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:06.029+0000 7ffbfc2847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:06.024 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:06.041+0000 7ffbfc2847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:06.026 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:06.041+0000 7ffbfc2847c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:06.026 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:06.041+0000 7ffbfc2847c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:36:06.027 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:06.041+0000 7ffbfc2847c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:36:06.027 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:06.041+0000 7ffbfc2847c0 -1 bluestore(td/repeer-on-acting-back/0/block) _read_bdev_label failed to open td/repeer-on-acting-back/0/block: (2) No such file or directory 2024-04-25T09:36:06.065 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:06.081+0000 7ffbfc2847c0 -1 bluestore(td/repeer-on-acting-back/0) _read_fsid unparsable uuid 2024-04-25T09:36:12.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/0/keyring 2024-04-25T09:36:12.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:36:12.441 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:36:12.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:36:12.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:36:13.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:36:13.010 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:36:13.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/0 --osd-journal=td/repeer-on-acting-back/0/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:36:13.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:36:13.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:36:13.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:36:13.047 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:13.061+0000 7f2febcb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:13.081 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:13.097+0000 7f2febcb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:13.083 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:13.097+0000 7f2febcb47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:13.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:36:13.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:36:13.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:36:13.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:36:13.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:36:13.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:13.451 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:36:13.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:36:13.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:13.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:13.663 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:13.677+0000 7f2febcb47c0 -1 Falling back to public interface 2024-04-25T09:36:13.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:14.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:14.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:14.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:36:14.917 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:36:14.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:14.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:15.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:16.282 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:16.297+0000 7f2febcb47c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:36:16.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:16.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:16.390 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:36:16.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:36:16.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:16.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:17.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:18.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:18.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:18.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:36:18.098 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:36:18.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:18.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:18.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:19.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:19.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:19.590 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:36:19.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:36:19.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:19.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:20.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:21.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:21.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:21.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:36:21.033 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:36:21.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:21.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:21.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:22.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:22.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:22.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:36:22.543 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:36:22.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:22.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:22.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:23.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:23.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:23.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:36:23.995 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:36:23.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:23.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:24.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:25.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:36:25.556 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:36:25.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:25.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:26.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:27.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:27.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:27.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:36:27.015 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:36:27.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:27.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:27.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:28.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:28.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:28.522 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:36:28.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:36:28.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:28.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:28.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:29.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:29.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:29.969 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:36:29.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:36:29.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:29.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:30.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:31.744 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:31.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:31.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:36:31.745 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:36:31.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:31.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:32.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:33.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:33.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:33.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:36:33.513 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:36:33.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:33.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:34.332 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:35.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:35.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:35.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:36:35.335 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:36:35.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:35.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:36.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:37.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:37.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:37.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:36:37.084 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:36:37.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:37.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:38.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:39.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:39.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:39.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:36:39.248 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:36:39.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:39.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:41.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:41.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:41.175 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:36:41.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:36:41.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:41.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:36:41.665 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4196357279,v1:127.0.0.1:6803/4196357279] [v2:127.0.0.1:6804/4196357279,v1:127.0.0.1:6805/4196357279] exists,up 824909d2-e965-4842-a8da-9ff7d5dc55a2 2024-04-25T09:36:41.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:36:41.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:36:41.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 1 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/1 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/1' 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/1/journal' 2024-04-25T09:36:41.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:36:41.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:36:41.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:36:41.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:36:41.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:36:41.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:36:41.669 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:36:41.669 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:36:41.669 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:36:41.669 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:36:41.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/1 2024-04-25T09:36:41.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:36:41.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f9ef971f-decd-4919-a897-f087a7aa47d1 2024-04-25T09:36:41.672 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 f9ef971f-decd-4919-a897-f087a7aa47d1 2024-04-25T09:36:41.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f9ef971f-decd-4919-a897-f087a7aa47d1' 2024-04-25T09:36:41.673 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:36:41.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQApJCpmCDgjKhAAlifUn69/GQMO9hTjFiwiJQ== 2024-04-25T09:36:41.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQApJCpmCDgjKhAAlifUn69/GQMO9hTjFiwiJQ=="}' 2024-04-25T09:36:41.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f9ef971f-decd-4919-a897-f087a7aa47d1 -i td/repeer-on-acting-back/1/new.json 2024-04-25T09:36:42.195 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:36:42.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/1/new.json 2024-04-25T09:36:42.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQApJCpmCDgjKhAAlifUn69/GQMO9hTjFiwiJQ== --osd-uuid f9ef971f-decd-4919-a897-f087a7aa47d1 2024-04-25T09:36:42.246 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:42.260+0000 7f280562f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:42.284 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:42.296+0000 7f280562f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:42.286 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:42.300+0000 7f280562f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:42.286 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:42.300+0000 7f280562f7c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:36:42.286 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:42.300+0000 7f280562f7c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:36:42.286 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:42.300+0000 7f280562f7c0 -1 bluestore(td/repeer-on-acting-back/1/block) _read_bdev_label failed to open td/repeer-on-acting-back/1/block: (2) No such file or directory 2024-04-25T09:36:42.409 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:42.420+0000 7f280562f7c0 -1 bluestore(td/repeer-on-acting-back/1) _read_fsid unparsable uuid 2024-04-25T09:36:48.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/1/keyring 2024-04-25T09:36:48.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:36:48.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:36:48.798 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:36:48.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:36:49.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:36:49.371 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:36:49.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/1 --osd-journal=td/repeer-on-acting-back/1/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:36:49.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:36:49.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:36:49.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:36:49.408 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:49.420+0000 7faa578597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:49.421 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:49.432+0000 7faa578597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:49.422 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:49.436+0000 7faa578597c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:36:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:36:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:36:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:36:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:36:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:36:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:49.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:36:49.774 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:36:49.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:49.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:49.968 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:49.980+0000 7faa578597c0 -1 Falling back to public interface 2024-04-25T09:36:50.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:51.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:51.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:51.179 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:36:51.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:36:51.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:51.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:51.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:52.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:52.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:52.606 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:36:52.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:36:52.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:52.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:52.798 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:36:52.808+0000 7faa578597c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:36:53.088 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:54.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:54.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:54.090 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:36:54.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:36:54.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:54.091 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:54.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:55.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:55.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:55.541 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:36:55.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:36:55.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:55.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:56.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:57.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:57.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:57.111 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:36:57.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:36:57.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:57.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:57.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:36:58.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:36:58.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:36:58.581 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:36:58.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:36:58.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:36:58.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:36:59.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:00.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:00.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:00.115 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:37:00.116 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:37:00.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:00.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:00.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:01.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:01.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:01.537 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:37:01.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:37:01.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:01.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:02.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:03.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:03.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:03.070 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:37:03.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:37:03.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:03.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:03.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:04.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:04.594 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:04.594 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:37:04.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:37:04.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:04.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:05.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:06.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:06.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:06.131 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:37:06.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:37:06.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:06.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:06.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:07.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:07.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:07.655 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:37:07.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:37:07.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:07.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:08.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:09.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:09.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:09.667 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:37:09.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:37:09.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:09.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:10.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:11.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:11.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:11.569 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:37:11.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:37:11.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:11.570 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:12.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:13.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:13.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:13.416 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:37:13.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:37:13.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:13.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:14.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:15.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:15.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:15.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:37:15.252 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:37:15.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:15.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:15.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:16.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:16.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:16.884 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:37:16.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:37:16.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:16.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:17.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:18.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:18.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:18.597 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:37:18.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:37:18.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:18.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:37:19.037 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/377541012,v1:127.0.0.1:6811/377541012] [v2:127.0.0.1:6812/377541012,v1:127.0.0.1:6813/377541012] exists,up f9ef971f-decd-4919-a897-f087a7aa47d1 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 2 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:37:19.040 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/2 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:37:19.041 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:37:19.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/2 2024-04-25T09:37:19.043 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:37:19.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7fd681d3-a520-4bbe-81d4-80ad095ddfd6 2024-04-25T09:37:19.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 7fd681d3-a520-4bbe-81d4-80ad095ddfd6' 2024-04-25T09:37:19.043 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 7fd681d3-a520-4bbe-81d4-80ad095ddfd6 2024-04-25T09:37:19.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:37:19.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBPJCpmsMX/BBAAOlynCP4o/e5kln1NO/w7CA== 2024-04-25T09:37:19.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBPJCpmsMX/BBAAOlynCP4o/e5kln1NO/w7CA=="}' 2024-04-25T09:37:19.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7fd681d3-a520-4bbe-81d4-80ad095ddfd6 -i td/repeer-on-acting-back/2/new.json 2024-04-25T09:37:19.590 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:37:19.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/2/new.json 2024-04-25T09:37:19.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBPJCpmsMX/BBAAOlynCP4o/e5kln1NO/w7CA== --osd-uuid 7fd681d3-a520-4bbe-81d4-80ad095ddfd6 2024-04-25T09:37:19.647 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:19.660+0000 7f6eca0197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:19.664 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:19.676+0000 7f6eca0197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:19.665 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:19.676+0000 7f6eca0197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:19.666 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:19.676+0000 7f6eca0197c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:37:19.666 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:19.676+0000 7f6eca0197c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:37:19.666 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:19.676+0000 7f6eca0197c0 -1 bluestore(td/repeer-on-acting-back/2/block) _read_bdev_label failed to open td/repeer-on-acting-back/2/block: (2) No such file or directory 2024-04-25T09:37:19.704 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:19.716+0000 7f6eca0197c0 -1 bluestore(td/repeer-on-acting-back/2) _read_fsid unparsable uuid 2024-04-25T09:37:26.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/2/keyring 2024-04-25T09:37:26.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:37:26.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:37:26.687 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:37:26.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:37:27.252 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:37:27.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:37:27.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:37:27.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:37:27.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:37:27.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:37:27.292 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:27.303+0000 7fd8a9b037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:27.329 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:27.339+0000 7fd8a9b037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:27.331 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:27.339+0000 7fd8a9b037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:27.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:37:27.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:37:27.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:37:27.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:37:27.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:37:27.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:27.728 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:37:27.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:37:27.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:27.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:27.904 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:27.915+0000 7fd8a9b037c0 -1 Falling back to public interface 2024-04-25T09:37:28.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:29.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:29.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:29.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:37:29.148 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:37:29.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:29.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:29.563 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:30.546 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:30.555+0000 7fd8a9b037c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:37:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:30.565 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:37:30.565 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:37:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:30.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:31.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:32.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:32.371 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:32.372 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:37:32.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:37:32.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:32.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:32.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:33.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:33.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:33.864 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:37:33.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:37:33.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:33.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:34.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:35.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:35.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:35.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:37:35.293 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:37:35.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:35.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:35.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:36.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:36.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:36.783 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:37:36.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:37:36.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:36.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:37.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:38.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:38.358 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:38.359 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:37:38.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:37:38.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:38.359 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:39.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:40.075 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:37:40.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:40.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:40.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:37:40.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:40.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:40.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:41.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:41.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:41.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:37:41.807 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:37:41.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:41.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:42.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:43.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:43.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:43.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:37:43.234 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:37:43.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:43.235 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:43.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:44.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:44.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:44.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:37:44.659 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:37:44.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:44.660 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:45.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:46.442 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:46.443 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:37:46.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:46.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:37:46.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:46.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:47.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:48.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:48.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:48.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:37:48.727 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:37:48.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:48.728 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:49.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:50.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:50.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:50.667 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:37:50.668 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:37:50.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:50.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:52.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:53.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:53.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:53.180 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:37:53.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:37:53.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:53.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:54.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:55.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:55.200 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:55.201 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:37:55.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:37:55.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:55.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:56.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:37:57.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:37:57.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:37:57.248 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:37:57.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:37:57.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:37:57.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:37:57.676 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3059439378,v1:127.0.0.1:6819/3059439378] [v2:127.0.0.1:6820/3059439378,v1:127.0.0.1:6821/3059439378] exists,up 7fd681d3-a520-4bbe-81d4-80ad095ddfd6 2024-04-25T09:37:57.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:37:57.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 3 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/3 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/3' 2024-04-25T09:37:57.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/3/journal' 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:37:57.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:37:57.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/3 2024-04-25T09:37:57.684 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:37:57.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ff4cbee-67e0-4be7-b5d6-ae7a4b4a364c 2024-04-25T09:37:57.684 INFO:tasks.workunit.client.0.smithi134.stdout:add osd3 6ff4cbee-67e0-4be7-b5d6-ae7a4b4a364c 2024-04-25T09:37:57.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 6ff4cbee-67e0-4be7-b5d6-ae7a4b4a364c' 2024-04-25T09:37:57.684 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:37:57.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1JCpmOy2wKhAAYatkDiT3Q77YP3OF3H6wWw== 2024-04-25T09:37:57.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1JCpmOy2wKhAAYatkDiT3Q77YP3OF3H6wWw=="}' 2024-04-25T09:37:57.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ff4cbee-67e0-4be7-b5d6-ae7a4b4a364c -i td/repeer-on-acting-back/3/new.json 2024-04-25T09:37:58.223 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:37:58.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/3/new.json 2024-04-25T09:37:58.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1JCpmOy2wKhAAYatkDiT3Q77YP3OF3H6wWw== --osd-uuid 6ff4cbee-67e0-4be7-b5d6-ae7a4b4a364c 2024-04-25T09:37:58.280 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:58.291+0000 7f9e28cfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:58.372 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:58.383+0000 7f9e28cfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:58.374 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:58.383+0000 7f9e28cfb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:37:58.374 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:58.383+0000 7f9e28cfb7c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:37:58.375 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:58.383+0000 7f9e28cfb7c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:37:58.375 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:58.383+0000 7f9e28cfb7c0 -1 bluestore(td/repeer-on-acting-back/3/block) _read_bdev_label failed to open td/repeer-on-acting-back/3/block: (2) No such file or directory 2024-04-25T09:37:58.496 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:37:58.507+0000 7f9e28cfb7c0 -1 bluestore(td/repeer-on-acting-back/3) _read_fsid unparsable uuid 2024-04-25T09:38:05.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/3/keyring 2024-04-25T09:38:05.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:38:05.192 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd3 key to auth repository 2024-04-25T09:38:05.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-04-25T09:38:05.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:38:05.748 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.3 2024-04-25T09:38:05.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-04-25T09:38:05.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:38:05.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:38:05.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/3 --osd-journal=td/repeer-on-acting-back/3/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:38:05.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:38:05.787 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:05.794+0000 7fc7588c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:05.828 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:05.838+0000 7fc7588c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:05.829 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:05.838+0000 7fc7588c27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:06.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-04-25T09:38:06.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:38:06.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-04-25T09:38:06.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:38:06.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:38:06.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:06.135 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:38:06.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:38:06.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:06.136 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:06.386 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:06.394+0000 7fc7588c27c0 -1 Falling back to public interface 2024-04-25T09:38:06.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:07.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:07.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:07.601 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:38:07.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:38:07.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:07.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:08.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:09.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:09.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:09.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:38:09.022 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:38:09.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:09.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:09.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:09.082+0000 7fc7588c27c0 -1 osd.3 0 log_to_monitors true 2024-04-25T09:38:09.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:10.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:10.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:10.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:38:10.487 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:38:10.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:10.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:11.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:12.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:12.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:12.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:38:12.056 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:38:12.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:12.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:12.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:13.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:13.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:13.571 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:38:13.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:38:13.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:13.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:14.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:15.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:15.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:15.132 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:38:15.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:38:15.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:15.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:15.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:16.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:16.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:16.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:38:16.586 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:38:16.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:16.586 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:17.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:18.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:18.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:18.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:38:18.102 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:38:18.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:18.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:18.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:19.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:19.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:19.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:38:19.522 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:38:19.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:19.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:20.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:21.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:21.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:21.008 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:38:21.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:38:21.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:21.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:21.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:22.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:22.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:22.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:38:22.504 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:38:22.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:22.505 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:22.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:23.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:23.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:23.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:38:23.955 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:38:23.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:23.956 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:25.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:26.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:26.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:26.298 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:38:26.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:38:26.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:26.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:27.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:28.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:28.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:28.470 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:38:28.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:38:28.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:28.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:29.196 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:30.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:30.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:30.198 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:38:30.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:38:30.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:30.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:30.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:31.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:31.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:31.762 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:38:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:38:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:31.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:32.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:33.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:33.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:33.323 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:38:33.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:38:33.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:33.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:34.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:35.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:35.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:35.058 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:38:35.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:38:35.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:35.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-04-25T09:38:35.494 INFO:tasks.workunit.client.0.smithi134.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/1295884719,v1:127.0.0.1:6827/1295884719] [v2:127.0.0.1:6828/1295884719,v1:127.0.0.1:6829/1295884719] exists,up 6ff4cbee-67e0-4be7-b5d6-ae7a4b4a364c 2024-04-25T09:38:35.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:38:35.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:38:35.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:38:35.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:38:35.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 4 2024-04-25T09:38:35.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:38:35.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=4 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/4 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/4' 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/4/journal' 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:38:35.499 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:38:35.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:38:35.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:38:35.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:38:35.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/4 2024-04-25T09:38:35.501 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:38:35.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a575e268-e1e4-4da6-a6ef-90e3a04c3b2c 2024-04-25T09:38:35.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd4 a575e268-e1e4-4da6-a6ef-90e3a04c3b2c' 2024-04-25T09:38:35.502 INFO:tasks.workunit.client.0.smithi134.stdout:add osd4 a575e268-e1e4-4da6-a6ef-90e3a04c3b2c 2024-04-25T09:38:35.502 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:38:35.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCbJCpmY9o3IBAAYE2trMTTk1A6h/1XE64izQ== 2024-04-25T09:38:35.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCbJCpmY9o3IBAAYE2trMTTk1A6h/1XE64izQ=="}' 2024-04-25T09:38:35.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a575e268-e1e4-4da6-a6ef-90e3a04c3b2c -i td/repeer-on-acting-back/4/new.json 2024-04-25T09:38:36.058 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:38:36.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/4/new.json 2024-04-25T09:38:36.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 4 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbJCpmY9o3IBAAYE2trMTTk1A6h/1XE64izQ== --osd-uuid a575e268-e1e4-4da6-a6ef-90e3a04c3b2c 2024-04-25T09:38:36.123 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:36.130+0000 7fcf3bce07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:36.193 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:36.202+0000 7fcf3bce07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:36.195 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:36.202+0000 7fcf3bce07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:36.196 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:36.202+0000 7fcf3bce07c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:38:36.196 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:36.202+0000 7fcf3bce07c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:38:36.196 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:36.202+0000 7fcf3bce07c0 -1 bluestore(td/repeer-on-acting-back/4/block) _read_bdev_label failed to open td/repeer-on-acting-back/4/block: (2) No such file or directory 2024-04-25T09:38:36.247 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:36.254+0000 7fcf3bce07c0 -1 bluestore(td/repeer-on-acting-back/4) _read_fsid unparsable uuid 2024-04-25T09:38:42.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/4/keyring 2024-04-25T09:38:42.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:38:42.959 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd4 key to auth repository 2024-04-25T09:38:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd4 key to auth repository 2024-04-25T09:38:42.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:38:43.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.4 2024-04-25T09:38:43.524 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.4 2024-04-25T09:38:43.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 4 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/4 --osd-journal=td/repeer-on-acting-back/4/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:38:43.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:38:43.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:38:43.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:38:43.555 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:43.566+0000 7fb7a93f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:43.557 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:43.566+0000 7fb7a93f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:43.559 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:43.566+0000 7fb7a93f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:38:43.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 4 2024-04-25T09:38:43.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:38:43.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=4 2024-04-25T09:38:43.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:38:43.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:38:43.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:43.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:38:43.996 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:38:43.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:43.996 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:44.130 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:44.142+0000 7fb7a93f37c0 -1 Falling back to public interface 2024-04-25T09:38:44.434 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:45.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:45.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:45.437 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:38:45.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:38:45.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:45.437 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:45.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:46.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:46.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:46.862 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:38:46.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:38:46.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:46.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:46.992 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:38:47.002+0000 7fb7a93f37c0 -1 osd.4 0 log_to_monitors true 2024-04-25T09:38:47.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:48.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:48.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:48.377 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:38:48.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:38:48.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:48.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:48.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:49.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:49.930 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:38:49.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:49.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:38:49.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:49.931 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:50.373 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:51.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:51.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:51.376 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:38:51.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:38:51.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:51.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:51.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:52.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:52.809 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:52.809 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:38:52.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:38:52.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:52.810 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:53.333 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:54.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:54.335 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:54.335 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:38:54.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:38:54.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:54.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:54.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:55.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:55.945 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:55.945 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:38:55.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:38:55.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:55.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:56.372 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:57.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:57.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:57.375 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:38:57.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:38:57.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:57.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:57.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:38:58.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:38:58.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:38:58.919 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:38:58.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:38:58.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:38:58.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:38:59.338 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:00.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:00.340 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:00.340 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:39:00.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:39:00.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:00.341 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:39:00.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:02.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:02.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:02.001 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:39:02.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:39:02.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:02.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:39:02.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:03.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:03.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:03.724 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:39:03.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:39:03.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:03.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:39:04.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:05.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:05.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:05.476 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:39:05.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:39:05.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:05.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:39:06.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:07.311 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:07.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:07.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:39:07.312 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:39:07.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:07.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:39:08.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:09.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:09.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:09.280 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:39:09.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:39:09.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:09.280 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:39:10.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:11.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:11.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:11.188 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:39:11.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:39:11.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:11.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:39:12.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:13.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:13.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:13.183 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:39:13.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:39:13.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:13.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.4 up' 2024-04-25T09:39:13.860 INFO:tasks.workunit.client.0.smithi134.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/1843393041,v1:127.0.0.1:6835/1843393041] [v2:127.0.0.1:6836/1843393041,v1:127.0.0.1:6837/1843393041] exists,up a575e268-e1e4-4da6-a6ef-90e3a04c3b2c 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:55: TEST_repeer_on_down_acting_member_coming_back: for i in $osds 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:57: TEST_repeer_on_down_acting_member_coming_back: run_osd td/repeer-on-acting-back 5 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=5 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repeer-on-acting-back/5 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:39:13.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/5' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/5/journal' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:39:13.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:39:13.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repeer-on-acting-back/5 2024-04-25T09:39:13.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:39:13.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8320a93c-7156-40c3-b955-55bd3104bf8d 2024-04-25T09:39:13.867 INFO:tasks.workunit.client.0.smithi134.stdout:add osd5 8320a93c-7156-40c3-b955-55bd3104bf8d 2024-04-25T09:39:13.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd5 8320a93c-7156-40c3-b955-55bd3104bf8d' 2024-04-25T09:39:13.867 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:39:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDBJCpmiULhNRAAofT4yQn2chUnxY1sneQS7g== 2024-04-25T09:39:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDBJCpmiULhNRAAofT4yQn2chUnxY1sneQS7g=="}' 2024-04-25T09:39:13.893 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8320a93c-7156-40c3-b955-55bd3104bf8d -i td/repeer-on-acting-back/5/new.json 2024-04-25T09:39:14.680 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:39:14.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repeer-on-acting-back/5/new.json 2024-04-25T09:39:14.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 5 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDBJCpmiULhNRAAofT4yQn2chUnxY1sneQS7g== --osd-uuid 8320a93c-7156-40c3-b955-55bd3104bf8d 2024-04-25T09:39:14.741 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:14.749+0000 7f51e443d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:39:14.762 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:14.773+0000 7f51e443d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:39:14.764 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:14.773+0000 7f51e443d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:39:14.764 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:14.773+0000 7f51e443d7c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:39:14.764 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:14.773+0000 7f51e443d7c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:39:14.764 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:14.773+0000 7f51e443d7c0 -1 bluestore(td/repeer-on-acting-back/5/block) _read_bdev_label failed to open td/repeer-on-acting-back/5/block: (2) No such file or directory 2024-04-25T09:39:14.887 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:14.897+0000 7f51e443d7c0 -1 bluestore(td/repeer-on-acting-back/5) _read_fsid unparsable uuid 2024-04-25T09:39:21.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repeer-on-acting-back/5/keyring 2024-04-25T09:39:21.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:39:21.656 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd5 key to auth repository 2024-04-25T09:39:21.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd5 key to auth repository 2024-04-25T09:39:21.656 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repeer-on-acting-back/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:39:22.227 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.5 2024-04-25T09:39:22.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.5 2024-04-25T09:39:22.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 5 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/5 --osd-journal=td/repeer-on-acting-back/5/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:39:22.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:39:22.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:39:22.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:39:22.260 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:22.269+0000 7f92b7f307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:39:22.289 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:22.297+0000 7f92b7f307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:39:22.291 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:22.301+0000 7f92b7f307c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:39:22.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 5 2024-04-25T09:39:22.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:39:22.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=5 2024-04-25T09:39:22.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:39:22.642 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:39:22.642 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:39:22.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:22.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:39:22.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:22.643 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:22.848 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:22.857+0000 7f92b7f307c0 -1 Falling back to public interface 2024-04-25T09:39:23.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:24.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:24.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:24.057 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:39:24.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:39:24.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:24.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:24.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:25.490 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:39:25.501+0000 7f92b7f307c0 -1 osd.5 0 log_to_monitors true 2024-04-25T09:39:25.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:25.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:25.503 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:39:25.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:39:25.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:25.503 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:26.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:27.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:27.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:27.215 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:39:27.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:39:27.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:27.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:27.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:28.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:28.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:28.671 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:39:28.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:39:28.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:28.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:29.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:30.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:30.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:30.142 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:39:30.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:39:30.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:30.142 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:30.792 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:31.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:31.794 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:39:31.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:31.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:39:31.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:31.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:32.450 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:33.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:33.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:33.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:39:33.452 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:39:33.452 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:33.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:33.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:34.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:34.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:34.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:39:34.886 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:39:34.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:34.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:35.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:36.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:36.417 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:36.417 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:39:36.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:39:36.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:36.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:36.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:37.833 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:37.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:37.834 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:39:37.834 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:39:37.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:37.835 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:38.363 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:39.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:39.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:39.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:39:39.365 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:39:39.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:39.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:39.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:40.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:40.855 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:40.856 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:39:40.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:39:40.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:40.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:42.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:43.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:43.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:43.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:39:43.212 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:39:43.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:43.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:43.966 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:44.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:44.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:44.968 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:39:44.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:39:44.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:44.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:45.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:46.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:46.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:46.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:39:46.856 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:39:46.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:46.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:47.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:48.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:48.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:48.553 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:39:48.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:39:48.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:48.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:49.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:50.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:50.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:50.675 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:39:50.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:39:50.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:50.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:51.215 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:39:52.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:39:52.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:39:52.217 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:39:52.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:39:52.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:39:52.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.5 up' 2024-04-25T09:39:52.654 INFO:tasks.workunit.client.0.smithi134.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/2395000130,v1:127.0.0.1:6843/2395000130] [v2:127.0.0.1:6844/2395000130,v1:127.0.0.1:6845/2395000130] exists,up 8320a93c-7156-40c3-b955-55bd3104bf8d 2024-04-25T09:39:52.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:39:52.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:39:52.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:39:52.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:60: TEST_repeer_on_down_acting_member_coming_back: create_pool test 1 1 2024-04-25T09:39:52.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 1 1 2024-04-25T09:39:53.310 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T09:39:53.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-04-25T09:39:54.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:61: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test size 3 2024-04-25T09:39:55.118 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 size to 3 2024-04-25T09:39:55.140 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:62: TEST_repeer_on_down_acting_member_coming_back: ceph osd pool set test min_size 2 2024-04-25T09:39:55.796 INFO:tasks.workunit.client.0.smithi134.stderr:set pool 1 min_size to 2 2024-04-25T09:39:55.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: ceph pg dump pools -f json 2024-04-25T09:39:55.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq .pool_stats 2024-04-25T09:39:55.819 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: jq '.[].poolid' 2024-04-25T09:39:56.225 INFO:tasks.workunit.client.0.smithi134.stderr:dumped pools 2024-04-25T09:39:56.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:63: TEST_repeer_on_down_acting_member_coming_back: local poolid=1 2024-04-25T09:39:56.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:64: TEST_repeer_on_down_acting_member_coming_back: local pgid=1.0 2024-04-25T09:39:56.243 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:67: TEST_repeer_on_down_acting_member_coming_back: ceph osd set-require-min-compat-client luminous 2024-04-25T09:39:56.908 INFO:tasks.workunit.client.0.smithi134.stderr:set require_min_compat_client to luminous 2024-04-25T09:39:56.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:69: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 2 3 2024-04-25T09:39:57.939 INFO:tasks.workunit.client.0.smithi134.stderr:set 1.0 pg_upmap mapping to [1,2,3] 2024-04-25T09:39:57.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:71: TEST_repeer_on_down_acting_member_coming_back: flush_pg_stats 2024-04-25T09:39:57.962 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:39:57.962 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:39:58.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:39:58.426 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:39:58.426 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:39:58.426 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T09:39:58.426 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T09:39:58.426 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T09:39:58.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:39:58.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:39:58.427 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:39:58.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836530 2024-04-25T09:39:58.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836530 2024-04-25T09:39:58.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530' 2024-04-25T09:39:58.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:39:58.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:39:58.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673002 2024-04-25T09:39:58.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673002 2024-04-25T09:39:58.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673002' 2024-04-25T09:39:58.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:39:58.812 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:39:59.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509471 2024-04-25T09:39:59.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509471 2024-04-25T09:39:59.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673002 2-64424509471' 2024-04-25T09:39:59.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:39:59.062 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:39:59.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345947 2024-04-25T09:39:59.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345947 2024-04-25T09:39:59.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673002 2-64424509471 3-85899345947' 2024-04-25T09:39:59.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:39:59.217 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:39:59.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182415 2024-04-25T09:39:59.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182415 2024-04-25T09:39:59.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673002 2-64424509471 3-85899345947 4-107374182415' 2024-04-25T09:39:59.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:39:59.370 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:39:59.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018891 2024-04-25T09:39:59.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018891 2024-04-25T09:39:59.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836530 1-42949673002 2-64424509471 3-85899345947 4-107374182415 5-128849018891' 2024-04-25T09:39:59.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:39:59.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836530 2024-04-25T09:39:59.525 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:39:59.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:39:59.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836530 2024-04-25T09:39:59.527 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:39:59.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836530 2024-04-25T09:39:59.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2024-04-25T09:39:59.528 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836530 2024-04-25T09:39:59.528 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:39:59.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836528 -lt 21474836530 2024-04-25T09:39:59.953 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:40:00.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:40:00.955 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:40:01.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836530 -lt 21474836530 2024-04-25T09:40:01.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:01.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673002 2024-04-25T09:40:01.388 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:01.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:40:01.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673002 2024-04-25T09:40:01.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:01.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673002 2024-04-25T09:40:01.392 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673002 2024-04-25T09:40:01.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673002' 2024-04-25T09:40:01.393 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:40:01.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673003 -lt 42949673002 2024-04-25T09:40:01.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:01.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509471 2024-04-25T09:40:01.813 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:01.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:40:01.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509471 2024-04-25T09:40:01.817 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:01.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509471 2024-04-25T09:40:01.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509471' 2024-04-25T09:40:01.818 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509471 2024-04-25T09:40:01.818 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:40:02.221 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509471 -lt 64424509471 2024-04-25T09:40:02.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:02.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345947 2024-04-25T09:40:02.223 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:02.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:40:02.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345947 2024-04-25T09:40:02.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:02.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345947 2024-04-25T09:40:02.227 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345947 2024-04-25T09:40:02.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345947' 2024-04-25T09:40:02.227 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:40:02.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345948 -lt 85899345947 2024-04-25T09:40:02.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:02.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:02.670 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182415 2024-04-25T09:40:02.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:40:02.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182415 2024-04-25T09:40:02.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:02.673 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182415 2024-04-25T09:40:02.674 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182415 2024-04-25T09:40:02.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182415' 2024-04-25T09:40:02.674 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:40:03.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182416 -lt 107374182415 2024-04-25T09:40:03.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:03.109 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018891 2024-04-25T09:40:03.109 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:03.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:40:03.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018891 2024-04-25T09:40:03.112 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:03.113 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018891 2024-04-25T09:40:03.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018891 2024-04-25T09:40:03.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018891' 2024-04-25T09:40:03.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:40:03.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018892 -lt 128849018891 2024-04-25T09:40:03.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:72: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:40:03.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:40:03.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:40:03.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:40:03.549 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:40:03.550 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:40:03.550 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:40:03.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:40:03.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:40:03.551 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:40:03.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:40:03.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:40:03.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:40:03.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:40:03.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:40:03.655 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:40:04.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:40:04.072 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:40:04.072 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:40:04.072 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T09:40:04.072 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T09:40:04.072 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T09:40:04.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:40:04.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:04.073 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:40:04.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836533 2024-04-25T09:40:04.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836533 2024-04-25T09:40:04.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533' 2024-04-25T09:40:04.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:04.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:40:04.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673005 2024-04-25T09:40:04.396 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673005 2024-04-25T09:40:04.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533 1-42949673005' 2024-04-25T09:40:04.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:04.397 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:40:04.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509474 2024-04-25T09:40:04.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509474 2024-04-25T09:40:04.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533 1-42949673005 2-64424509474' 2024-04-25T09:40:04.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:04.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:40:04.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345951 2024-04-25T09:40:04.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345951 2024-04-25T09:40:04.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533 1-42949673005 2-64424509474 3-85899345951' 2024-04-25T09:40:04.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:04.692 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:40:04.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182419 2024-04-25T09:40:04.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182419 2024-04-25T09:40:04.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533 1-42949673005 2-64424509474 3-85899345951 4-107374182419' 2024-04-25T09:40:04.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:04.880 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:40:05.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018894 2024-04-25T09:40:05.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018894 2024-04-25T09:40:05.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836533 1-42949673005 2-64424509474 3-85899345951 4-107374182419 5-128849018894' 2024-04-25T09:40:05.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:05.044 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836533 2024-04-25T09:40:05.045 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:05.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:40:05.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836533 2024-04-25T09:40:05.046 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:05.047 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836533 2024-04-25T09:40:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836533 2024-04-25T09:40:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2024-04-25T09:40:05.048 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:40:05.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836533 -lt 21474836533 2024-04-25T09:40:05.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:05.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673005 2024-04-25T09:40:05.483 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:05.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:40:05.485 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673005 2024-04-25T09:40:05.485 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:05.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673005 2024-04-25T09:40:05.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673005' 2024-04-25T09:40:05.486 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673005 2024-04-25T09:40:05.487 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:40:05.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673003 -lt 42949673005 2024-04-25T09:40:05.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:40:06.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:40:06.917 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:40:07.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673006 -lt 42949673005 2024-04-25T09:40:07.345 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:07.346 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509474 2024-04-25T09:40:07.347 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:07.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:40:07.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509474 2024-04-25T09:40:07.349 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:07.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509474 2024-04-25T09:40:07.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509474' 2024-04-25T09:40:07.350 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509474 2024-04-25T09:40:07.351 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:40:07.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509474 -lt 64424509474 2024-04-25T09:40:07.787 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:07.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345951 2024-04-25T09:40:07.788 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:07.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:40:07.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345951 2024-04-25T09:40:07.790 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:07.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345951 2024-04-25T09:40:07.791 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345951 2024-04-25T09:40:07.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345951' 2024-04-25T09:40:07.792 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:40:08.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345951 -lt 85899345951 2024-04-25T09:40:08.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:08.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182419 2024-04-25T09:40:08.277 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:08.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:40:08.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182419 2024-04-25T09:40:08.280 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:08.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182419 2024-04-25T09:40:08.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182419' 2024-04-25T09:40:08.281 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182419 2024-04-25T09:40:08.282 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:40:08.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182419 -lt 107374182419 2024-04-25T09:40:08.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:08.733 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018894 2024-04-25T09:40:08.734 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:08.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:40:08.736 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018894 2024-04-25T09:40:08.736 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:08.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018894 2024-04-25T09:40:08.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018894' 2024-04-25T09:40:08.738 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018894 2024-04-25T09:40:08.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:40:09.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018895 -lt 128849018894 2024-04-25T09:40:09.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:40:09.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:40:09.207 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:40:09.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:40:09.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:40:09.749 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:40:09.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:40:09.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:40:09.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:40:09.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:40:09.750 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:40:10.182 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:40:10.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:40:10.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:40:10.183 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:40:10.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:40:10.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:40:10.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:40:10.750 INFO:tasks.workunit.client.0.smithi134.stdout:writing initial objects 2024-04-25T09:40:10.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:74: TEST_repeer_on_down_acting_member_coming_back: echo 'writing initial objects' 2024-04-25T09:40:10.751 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: seq 1 100 2024-04-25T09:40:10.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:10.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_1 /etc/fstab 2024-04-25T09:40:10.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:10.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_2 /etc/fstab 2024-04-25T09:40:11.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:11.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_3 /etc/fstab 2024-04-25T09:40:11.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:11.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_4 /etc/fstab 2024-04-25T09:40:11.493 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:11.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_5 /etc/fstab 2024-04-25T09:40:11.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:11.669 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_6 /etc/fstab 2024-04-25T09:40:11.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:11.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_7 /etc/fstab 2024-04-25T09:40:12.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:12.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_8 /etc/fstab 2024-04-25T09:40:12.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:12.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_9 /etc/fstab 2024-04-25T09:40:12.407 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:12.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_10 /etc/fstab 2024-04-25T09:40:12.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:12.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_11 /etc/fstab 2024-04-25T09:40:12.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:12.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_12 /etc/fstab 2024-04-25T09:40:13.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:13.001 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_13 /etc/fstab 2024-04-25T09:40:13.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:13.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_14 /etc/fstab 2024-04-25T09:40:13.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:13.344 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_15 /etc/fstab 2024-04-25T09:40:13.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:13.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_16 /etc/fstab 2024-04-25T09:40:13.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:13.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_17 /etc/fstab 2024-04-25T09:40:13.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:13.888 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_18 /etc/fstab 2024-04-25T09:40:14.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:14.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_19 /etc/fstab 2024-04-25T09:40:14.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:14.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_20 /etc/fstab 2024-04-25T09:40:14.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:14.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_21 /etc/fstab 2024-04-25T09:40:14.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:14.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_22 /etc/fstab 2024-04-25T09:40:14.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:14.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_23 /etc/fstab 2024-04-25T09:40:14.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:14.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_24 /etc/fstab 2024-04-25T09:40:15.102 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:15.103 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_25 /etc/fstab 2024-04-25T09:40:15.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:15.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_26 /etc/fstab 2024-04-25T09:40:15.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:15.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_27 /etc/fstab 2024-04-25T09:40:15.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:15.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_28 /etc/fstab 2024-04-25T09:40:15.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:15.764 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_29 /etc/fstab 2024-04-25T09:40:15.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:15.932 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_30 /etc/fstab 2024-04-25T09:40:16.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:16.108 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_31 /etc/fstab 2024-04-25T09:40:16.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:16.275 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_32 /etc/fstab 2024-04-25T09:40:16.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:16.451 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_33 /etc/fstab 2024-04-25T09:40:16.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:16.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_34 /etc/fstab 2024-04-25T09:40:16.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:16.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_35 /etc/fstab 2024-04-25T09:40:17.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:17.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_36 /etc/fstab 2024-04-25T09:40:17.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:17.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_37 /etc/fstab 2024-04-25T09:40:17.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:17.339 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_38 /etc/fstab 2024-04-25T09:40:17.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:17.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_39 /etc/fstab 2024-04-25T09:40:17.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:17.674 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_40 /etc/fstab 2024-04-25T09:40:17.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:17.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_41 /etc/fstab 2024-04-25T09:40:18.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:18.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_42 /etc/fstab 2024-04-25T09:40:18.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:18.193 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_43 /etc/fstab 2024-04-25T09:40:18.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:18.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_44 /etc/fstab 2024-04-25T09:40:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:18.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_45 /etc/fstab 2024-04-25T09:40:18.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:18.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_46 /etc/fstab 2024-04-25T09:40:18.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:18.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_47 /etc/fstab 2024-04-25T09:40:19.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:19.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_48 /etc/fstab 2024-04-25T09:40:19.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:19.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_49 /etc/fstab 2024-04-25T09:40:19.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:19.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_50 /etc/fstab 2024-04-25T09:40:19.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:19.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_51 /etc/fstab 2024-04-25T09:40:19.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:19.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_52 /etc/fstab 2024-04-25T09:40:19.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:19.927 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_53 /etc/fstab 2024-04-25T09:40:20.094 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:20.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_54 /etc/fstab 2024-04-25T09:40:20.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:20.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_55 /etc/fstab 2024-04-25T09:40:20.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:20.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_56 /etc/fstab 2024-04-25T09:40:20.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:20.589 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_57 /etc/fstab 2024-04-25T09:40:20.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:20.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_58 /etc/fstab 2024-04-25T09:40:20.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:20.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_59 /etc/fstab 2024-04-25T09:40:21.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:21.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_60 /etc/fstab 2024-04-25T09:40:21.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:21.317 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_61 /etc/fstab 2024-04-25T09:40:21.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:21.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_62 /etc/fstab 2024-04-25T09:40:21.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:21.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_63 /etc/fstab 2024-04-25T09:40:21.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:21.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_64 /etc/fstab 2024-04-25T09:40:22.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:22.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_65 /etc/fstab 2024-04-25T09:40:22.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:22.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_66 /etc/fstab 2024-04-25T09:40:22.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:22.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_67 /etc/fstab 2024-04-25T09:40:22.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:22.523 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_68 /etc/fstab 2024-04-25T09:40:22.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:22.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_69 /etc/fstab 2024-04-25T09:40:22.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:22.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_70 /etc/fstab 2024-04-25T09:40:23.042 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:23.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_71 /etc/fstab 2024-04-25T09:40:23.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:23.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_72 /etc/fstab 2024-04-25T09:40:23.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:23.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_73 /etc/fstab 2024-04-25T09:40:23.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:23.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_74 /etc/fstab 2024-04-25T09:40:23.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:23.712 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_75 /etc/fstab 2024-04-25T09:40:23.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:23.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_76 /etc/fstab 2024-04-25T09:40:24.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:24.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_77 /etc/fstab 2024-04-25T09:40:24.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:24.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_78 /etc/fstab 2024-04-25T09:40:24.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:24.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_79 /etc/fstab 2024-04-25T09:40:24.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:24.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_80 /etc/fstab 2024-04-25T09:40:24.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:24.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_81 /etc/fstab 2024-04-25T09:40:24.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:24.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_82 /etc/fstab 2024-04-25T09:40:25.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:25.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_83 /etc/fstab 2024-04-25T09:40:25.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:25.254 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_84 /etc/fstab 2024-04-25T09:40:25.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:25.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_85 /etc/fstab 2024-04-25T09:40:25.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:25.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_86 /etc/fstab 2024-04-25T09:40:25.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:25.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_87 /etc/fstab 2024-04-25T09:40:25.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:25.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_88 /etc/fstab 2024-04-25T09:40:26.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:26.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_89 /etc/fstab 2024-04-25T09:40:26.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:26.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_90 /etc/fstab 2024-04-25T09:40:26.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:26.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_91 /etc/fstab 2024-04-25T09:40:26.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:26.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_92 /etc/fstab 2024-04-25T09:40:26.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:26.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_93 /etc/fstab 2024-04-25T09:40:27.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:27.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_94 /etc/fstab 2024-04-25T09:40:27.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:27.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_95 /etc/fstab 2024-04-25T09:40:27.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:27.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_96 /etc/fstab 2024-04-25T09:40:27.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:27.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_97 /etc/fstab 2024-04-25T09:40:27.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:27.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_98 /etc/fstab 2024-04-25T09:40:27.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:27.849 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_99 /etc/fstab 2024-04-25T09:40:28.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:76: TEST_repeer_on_down_acting_member_coming_back: for i in $(seq 1 $testobjects) 2024-04-25T09:40:28.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:78: TEST_repeer_on_down_acting_member_coming_back: rados -p test put existing_100 /etc/fstab 2024-04-25T09:40:28.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T09:40:28.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:81: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:40:28.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:40:28.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:40:28.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:40:28.561 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T09:40:28.561 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:40:28.561 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:40:28.562 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:40:28.562 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:40:28.562 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:40:28.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T09:40:28.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:40:28.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:40:28.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:40:28.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:40:28.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:40:29.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:40:29.097 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:40:29.097 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:40:29.097 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T09:40:29.098 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T09:40:29.098 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T09:40:29.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:40:29.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:29.098 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:40:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836540 2024-04-25T09:40:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836540 2024-04-25T09:40:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540' 2024-04-25T09:40:29.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:29.286 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:40:29.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673013 2024-04-25T09:40:29.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673013 2024-04-25T09:40:29.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673013' 2024-04-25T09:40:29.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:29.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:40:29.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509481 2024-04-25T09:40:29.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509481 2024-04-25T09:40:29.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673013 2-64424509481' 2024-04-25T09:40:29.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:29.608 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:40:29.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345958 2024-04-25T09:40:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345958 2024-04-25T09:40:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673013 2-64424509481 3-85899345958' 2024-04-25T09:40:29.763 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:29.763 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:40:29.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182426 2024-04-25T09:40:29.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182426 2024-04-25T09:40:29.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673013 2-64424509481 3-85899345958 4-107374182426' 2024-04-25T09:40:29.913 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:29.913 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:40:30.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018901 2024-04-25T09:40:30.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018901 2024-04-25T09:40:30.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836540 1-42949673013 2-64424509481 3-85899345958 4-107374182426 5-128849018901' 2024-04-25T09:40:30.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:30.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836540 2024-04-25T09:40:30.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:30.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:40:30.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836540 2024-04-25T09:40:30.070 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:30.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836540 2024-04-25T09:40:30.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836540' 2024-04-25T09:40:30.071 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836540 2024-04-25T09:40:30.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:40:30.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836540 -lt 21474836540 2024-04-25T09:40:30.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:30.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673013 2024-04-25T09:40:30.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:30.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:40:30.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673013 2024-04-25T09:40:30.514 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:30.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673013 2024-04-25T09:40:30.515 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673013 2024-04-25T09:40:30.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673013' 2024-04-25T09:40:30.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:40:30.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673013 -lt 42949673013 2024-04-25T09:40:30.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:30.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509481 2024-04-25T09:40:30.949 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:30.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:40:30.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509481 2024-04-25T09:40:30.951 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:30.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509481 2024-04-25T09:40:30.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509481' 2024-04-25T09:40:30.952 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509481 2024-04-25T09:40:30.953 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:40:31.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509481 -lt 64424509481 2024-04-25T09:40:31.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:31.387 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345958 2024-04-25T09:40:31.387 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:31.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:40:31.389 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345958 2024-04-25T09:40:31.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:31.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345958 2024-04-25T09:40:31.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345958' 2024-04-25T09:40:31.391 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345958 2024-04-25T09:40:31.391 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:40:31.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345958 -lt 85899345958 2024-04-25T09:40:31.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:31.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182426 2024-04-25T09:40:31.798 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:31.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:40:31.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182426 2024-04-25T09:40:31.800 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:31.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182426 2024-04-25T09:40:31.801 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182426 2024-04-25T09:40:31.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182426' 2024-04-25T09:40:31.802 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:40:32.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182426 -lt 107374182426 2024-04-25T09:40:32.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:32.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018901 2024-04-25T09:40:32.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:32.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:40:32.260 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018901 2024-04-25T09:40:32.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:32.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018901 2024-04-25T09:40:32.262 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018901 2024-04-25T09:40:32.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018901' 2024-04-25T09:40:32.262 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:40:32.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018902 -lt 128849018901 2024-04-25T09:40:32.671 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:40:32.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:40:32.672 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:40:33.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:40:33.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:40:33.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:40:33.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:40:33.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:40:33.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:40:33.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:40:33.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:40:33.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:40:33.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:40:33.690 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:40:33.690 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:40:34.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:40:34.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:40:34.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:40:34.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:84: TEST_repeer_on_down_acting_member_coming_back: ceph osd pg-upmap 1.0 1 4 5 2024-04-25T09:40:35.157 INFO:tasks.workunit.client.0.smithi134.stderr:set 1.0 pg_upmap mapping to [1,4,5] 2024-04-25T09:40:35.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:87: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:40:37.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:40:37.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:40:37.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:90: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:40:37.360 INFO:tasks.workunit.client.0.smithi134.stdout: 2, 2024-04-25T09:40:37.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:93: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back KILL osd.2 2024-04-25T09:40:37.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:40:37.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:40:37.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:40:37.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:40:37.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:40:37.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:40:37.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:94: TEST_repeer_on_down_acting_member_coming_back: ceph osd down osd.2 2024-04-25T09:40:37.970 INFO:tasks.workunit.client.0.smithi134.stderr:osd.2 is already down. 2024-04-25T09:40:37.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:97: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:40:39.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:40:39.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:40:39.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:100: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:40:40.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:103: TEST_repeer_on_down_acting_member_coming_back: activate_osd td/repeer-on-acting-back 2 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repeer-on-acting-back 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repeer-on-acting-back/2 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repeer-on-acting-back/2' 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repeer-on-acting-back/2/journal' 2024-04-25T09:40:40.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repeer-on-acting-back' 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repeer-on-acting-back/$name.log' 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repeer-on-acting-back/$name.pid' 2024-04-25T09:40:40.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:40:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:40:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:40:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:40:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:40:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:40:40.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repeer-on-acting-back/2 2024-04-25T09:40:40.173 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:40:40.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-04-25T09:40:40.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=9f34e074-d708-4a57-abe7-655e4678fcd9 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repeer-on-acting-back/2 --osd-journal=td/repeer-on-acting-back/2/journal --chdir= --run-dir=td/repeer-on-acting-back '--admin-socket=/tmp/ceph-asok.478728/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repeer-on-acting-back/$name.log' '--pid-file=td/repeer-on-acting-back/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:40:40.174 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repeer-on-acting-back/2/whoami 2024-04-25T09:40:40.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-04-25T09:40:40.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:40:40.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:40:40.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:40:40.211 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:40:40.219+0000 7f54bc02e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:40.214 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:40:40.219+0000 7f54bc02e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:40.216 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:40:40.223+0000 7f54bc02e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:40:40.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-04-25T09:40:40.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:40:40.720 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:40:40.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:40:40.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:40:40.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:40.721 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:40:40.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:40:40.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:40.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:40:40.770 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:40:40.775+0000 7f54bc02e7c0 -1 Falling back to public interface 2024-04-25T09:40:41.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:42.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:42.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:42.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:40:42.161 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:40:42.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:42.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:40:42.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:43.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:43.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:43.621 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:40:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:40:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:43.622 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:40:44.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:45.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:45.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:45.066 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:40:45.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:40:45.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:45.067 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:40:45.074 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:40:45.079+0000 7f54bc02e7c0 -1 osd.2 46 log_to_monitors true 2024-04-25T09:40:45.731 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:40:46.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:40:46.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:46.733 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:40:46.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:40:46.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:46.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:40:47.352 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/2851472257,v1:127.0.0.1:6819/2851472257] [v2:127.0.0.1:6820/2851472257,v1:127.0.0.1:6821/2851472257] exists,up 7fd681d3-a520-4bbe-81d4-80ad095ddfd6 2024-04-25T09:40:47.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:40:47.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:40:47.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:40:47.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:104: TEST_repeer_on_down_acting_member_coming_back: wait_for_osd up 2 2024-04-25T09:40:47.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:40:47.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:40:47.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:40:47.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:40:47.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:40:47.354 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:40:47.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:40:47.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:40:47.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:40:47.794 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 50 up_thru 0 down_at 47 last_clean_interval [15,46) [v2:127.0.0.1:6818/2851472257,v1:127.0.0.1:6819/2851472257] [v2:127.0.0.1:6820/2851472257,v1:127.0.0.1:6821/2851472257] exists,up 7fd681d3-a520-4bbe-81d4-80ad095ddfd6 2024-04-25T09:40:47.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:40:47.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:40:47.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:40:47.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:107: TEST_repeer_on_down_acting_member_coming_back: sleep 2 2024-04-25T09:40:49.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: ceph pg 1.0 query 2024-04-25T09:40:49.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: jq .acting 2024-04-25T09:40:49.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:110: TEST_repeer_on_down_acting_member_coming_back: grep 2 2024-04-25T09:40:49.977 INFO:tasks.workunit.client.0.smithi134.stdout: 2 2024-04-25T09:40:49.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: WAIT_FOR_CLEAN_TIMEOUT=20 2024-04-25T09:40:49.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:112: TEST_repeer_on_down_acting_member_coming_back: wait_for_clean 2024-04-25T09:40:49.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:40:49.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:40:49.983 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:40:49.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 20 .1 2024-04-25T09:40:49.983 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:40:49.983 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:40:49.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:40:49.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:40:49.983 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:40:50.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '7.3') 2024-04-25T09:40:50.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:40:50.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:40:50.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:40:50.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:40:50.037 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:40:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:40:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:40:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:2 2024-04-25T09:40:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:3 2024-04-25T09:40:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:4 2024-04-25T09:40:50.521 INFO:tasks.workunit.client.0.smithi134.stderr:5' 2024-04-25T09:40:50.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:40:50.522 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:50.522 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:40:50.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836546 2024-04-25T09:40:50.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836546 2024-04-25T09:40:50.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836546' 2024-04-25T09:40:50.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:50.699 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:40:50.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=42949673019 2024-04-25T09:40:50.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 42949673019 2024-04-25T09:40:50.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836546 1-42949673019' 2024-04-25T09:40:50.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:50.863 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:40:51.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=214748364804 2024-04-25T09:40:51.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 214748364804 2024-04-25T09:40:51.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836546 1-42949673019 2-214748364804' 2024-04-25T09:40:51.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:51.031 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-04-25T09:40:51.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=85899345964 2024-04-25T09:40:51.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 85899345964 2024-04-25T09:40:51.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836546 1-42949673019 2-214748364804 3-85899345964' 2024-04-25T09:40:51.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:51.189 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2024-04-25T09:40:51.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=107374182432 2024-04-25T09:40:51.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 107374182432 2024-04-25T09:40:51.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836546 1-42949673019 2-214748364804 3-85899345964 4-107374182432' 2024-04-25T09:40:51.350 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:40:51.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2024-04-25T09:40:51.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=128849018907 2024-04-25T09:40:51.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 128849018907 2024-04-25T09:40:51.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836546 1-42949673019 2-214748364804 3-85899345964 4-107374182432 5-128849018907' 2024-04-25T09:40:51.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:51.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836546 2024-04-25T09:40:51.512 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:51.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:40:51.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836546 2024-04-25T09:40:51.513 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:51.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836546 2024-04-25T09:40:51.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836546' 2024-04-25T09:40:51.515 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836546 2024-04-25T09:40:51.515 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:40:51.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836544 -lt 21474836546 2024-04-25T09:40:51.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:40:52.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:40:52.935 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:40:53.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836547 -lt 21474836546 2024-04-25T09:40:53.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:53.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-42949673019 2024-04-25T09:40:53.375 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:53.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:40:53.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-42949673019 2024-04-25T09:40:53.377 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:53.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=42949673019 2024-04-25T09:40:53.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 42949673019' 2024-04-25T09:40:53.379 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 42949673019 2024-04-25T09:40:53.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:40:53.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 42949673019 -lt 42949673019 2024-04-25T09:40:53.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:53.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-214748364804 2024-04-25T09:40:53.988 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:53.990 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:40:53.990 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-214748364804 2024-04-25T09:40:53.991 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:53.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=214748364804 2024-04-25T09:40:53.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 214748364804' 2024-04-25T09:40:53.992 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 214748364804 2024-04-25T09:40:53.992 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:40:54.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 214748364804 -lt 214748364804 2024-04-25T09:40:54.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:54.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 3-85899345964 2024-04-25T09:40:54.637 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:54.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=3 2024-04-25T09:40:54.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 3-85899345964 2024-04-25T09:40:54.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:54.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=85899345964 2024-04-25T09:40:54.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.3 seq 85899345964' 2024-04-25T09:40:54.641 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.3 seq 85899345964 2024-04-25T09:40:54.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 3 2024-04-25T09:40:55.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 85899345964 -lt 85899345964 2024-04-25T09:40:55.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:55.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 4-107374182432 2024-04-25T09:40:55.157 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:55.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=4 2024-04-25T09:40:55.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 4-107374182432 2024-04-25T09:40:55.159 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:55.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=107374182432 2024-04-25T09:40:55.161 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.4 seq 107374182432 2024-04-25T09:40:55.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.4 seq 107374182432' 2024-04-25T09:40:55.161 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 4 2024-04-25T09:40:55.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 107374182432 -lt 107374182432 2024-04-25T09:40:55.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:40:55.683 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:40:55.683 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 5-128849018907 2024-04-25T09:40:55.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=5 2024-04-25T09:40:55.685 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 5-128849018907 2024-04-25T09:40:55.685 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:40:55.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=128849018907 2024-04-25T09:40:55.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.5 seq 128849018907' 2024-04-25T09:40:55.686 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.5 seq 128849018907 2024-04-25T09:40:55.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 5 2024-04-25T09:40:56.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 128849018908 -lt 128849018907 2024-04-25T09:40:56.225 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:40:56.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:40:56.226 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:40:56.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:40:56.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:40:56.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:40:56.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:40:56.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:40:56.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:40:56.885 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:40:56.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:40:57.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:40:57.435 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:40:57.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:40:57.436 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:40:58.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:40:58.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-04-25T09:40:58.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-04-25T09:40:58.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-04-25T09:40:58.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:40:58.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:40:58.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:40:58.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:40:58.134 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:40:58.134 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:40:58.134 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:40:58.134 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:40:58.134 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:40:58.134 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:40:58.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:40:58.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:40:58.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:40:58.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:40:59.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:40:59.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:40:59.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:40:59.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:40:59.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:40:59.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:40:59.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:40:59.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:40:59.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:40:59.951 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=2556 2024-04-25T09:40:59.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 2556 '!=' null 2024-04-25T09:40:59.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:40:59.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:40:59.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:41:00.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:41:00.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:41:00.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:41:00.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:41:00.053 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:41:00.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:41:00.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:41:00.054 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:41:00.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:41:00.476 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:41:00.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:41:00.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:41:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:41:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:41:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:41:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:41:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:41:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:41:01.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:41:01.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:41:01.056 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:41:01.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3088 2024-04-25T09:41:01.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3088 '!=' null 2024-04-25T09:41:01.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:41:01.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:41:01.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:41:01.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:41:01.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:41:01.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:41:01.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:41:01.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:41:01.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:41:01.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:41:01.679 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:41:02.090 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:41:02.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:41:02.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:41:02.091 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:41:02.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:41:02.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:41:02.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:41:02.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:41:02.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:41:02.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:41:02.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:41:02.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:41:02.621 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:41:03.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=3088 2024-04-25T09:41:03.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 3088 '!=' null 2024-04-25T09:41:03.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:41:03.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:41:03.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:41:03.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:41:03.255 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:41:03.255 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:41:03.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:41:03.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:41:03.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:41:03.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:41:03.256 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:41:03.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:41:03.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:41:03.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:41:03.659 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:41:04.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:41:04.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:41:04.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:41:04.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:41:04.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:41:04.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:41:04.247 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:41:04.248 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:41:04.248 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:41:04.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1544 2024-04-25T09:41:04.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1544 '!=' null 2024-04-25T09:41:04.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:41:04.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:41:04.839 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:41:04.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:41:04.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:41:04.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:41:04.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:41:04.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:41:04.941 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:41:04.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:41:04.942 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:41:05.349 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-04-25T09:41:05.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:41:05.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:41:05.350 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:41:05.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 1 2024-04-25T09:41:05.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-04-25T09:41:05.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-04-25T09:41:05.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-04-25T09:41:05.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-04-25T09:41:05.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-04-25T09:41:05.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-04-25T09:41:05.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-04-25T09:41:05.890 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-04-25T09:41:06.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=1544 2024-04-25T09:41:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test 1544 '!=' null 2024-04-25T09:41:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: loop=0 2024-04-25T09:41:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-04-25T09:41:06.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-04-25T09:41:06.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-04-25T09:41:06.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:41:06.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:41:06.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:41:06.532 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:41:06.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:41:06.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:41:06.533 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:41:06.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:41:06.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:41:06.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:41:06.939 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:41:07.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:41:07.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:41:07.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:41:07.486 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: find td/repeer-on-acting-back -name '*osd*log' 2024-04-25T09:41:07.487 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:114: TEST_repeer_on_down_acting_member_coming_back: grep -q 'Active: got notify from previous acting member.*, requesting pg_temp change' td/repeer-on-acting-back/osd.5.log td/repeer-on-acting-back/osd.3.log td/repeer-on-acting-back/osd.4.log td/repeer-on-acting-back/osd.2.log td/repeer-on-acting-back/osd.1.log td/repeer-on-acting-back/osd.0.log 2024-04-25T09:41:07.499 INFO:tasks.workunit.client.0.smithi134.stdout:success 2024-04-25T09:41:07.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:119: TEST_repeer_on_down_acting_member_coming_back: echo success 2024-04-25T09:41:07.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:121: TEST_repeer_on_down_acting_member_coming_back: delete_pool test 2024-04-25T09:41:07.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: delete_pool: local poolname=test 2024-04-25T09:41:07.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2024-04-25T09:41:08.211 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' does not exist 2024-04-25T09:41:08.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:122: TEST_repeer_on_down_acting_member_coming_back: kill_daemons td/repeer-on-acting-back 2024-04-25T09:41:08.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:41:08.228 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:41:08.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:41:08.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:41:08.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:41:09.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:41:09.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repeer-on-acting-back.sh:42: run: teardown td/repeer-on-acting-back 2024-04-25T09:41:09.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:41:09.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:41:09.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:41:09.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:41:09.543 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:41:09.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:41:09.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:41:09.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:41:09.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:41:09.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:41:09.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:41:09.553 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:41:09.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:41:09.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:41:09.555 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:41:09.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:41:09.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:41:09.556 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:41:09.556 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:41:09.557 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:41:09.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:41:09.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.478728 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/repeer-on-acting-back 0 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repeer-on-acting-back 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:41:09.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repeer-on-acting-back KILL 2024-04-25T09:41:09.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:41:09.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:41:09.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:41:09.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:41:09.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:41:09.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:41:09.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:41:09.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:41:09.600 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:41:09.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:41:09.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:41:09.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:41:09.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:41:09.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:41:09.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:41:09.603 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:41:09.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:41:09.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:41:09.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repeer-on-acting-back 2024-04-25T09:41:09.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:41:09.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:09.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.478728 2024-04-25T09:41:09.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.478728 2024-04-25T09:41:09.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:41:09.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:41:09.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:41:09.608 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:41:09.609 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:41:09.619 INFO:tasks.workunit:Running workunit osd/repro_long_log.sh... 2024-04-25T09:41:09.619 DEBUG:teuthology.orchestra.run.smithi134:workunit test osd/repro_long_log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=a08304379f921e5e229e92c75ede8800059313ff TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh 2024-04-25T09:41:09.669 INFO:tasks.workunit.client.0.smithi134.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-04-25T09:41:09.675 INFO:tasks.workunit.client.0.smithi134.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-04-25T09:41:09.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:41:09.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-04-25T09:41:09.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: export PYTHONWARNINGS=ignore 2024-04-25T09:41:09.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2324: main: PYTHONWARNINGS=ignore 2024-04-25T09:41:09.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: export CEPH_CONF=/dev/null 2024-04-25T09:41:09.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: main: CEPH_CONF=/dev/null 2024-04-25T09:41:09.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: main: unset CEPH_ARGS 2024-04-25T09:41:09.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2328: main: local code 2024-04-25T09:41:09.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2329: main: run td/repro-long-log 2024-04-25T09:41:09.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:23: run: local dir=td/repro-long-log 2024-04-25T09:41:09.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:24: run: shift 2024-04-25T09:41:09.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: export CEPH_MON=127.0.0.1:7100 2024-04-25T09:41:09.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:26: run: CEPH_MON=127.0.0.1:7100 2024-04-25T09:41:09.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:27: run: export CEPH_ARGS 2024-04-25T09:41:09.677 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: uuidgen 2024-04-25T09:41:09.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:28: run: CEPH_ARGS+='--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none ' 2024-04-25T09:41:09.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:29: run: CEPH_ARGS+='--mon-host=127.0.0.1:7100 ' 2024-04-25T09:41:09.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: set 2024-04-25T09:41:09.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-04-25T09:41:09.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:31: run: local 'funcs=TEST_repro_long_log1 2024-04-25T09:41:09.680 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_repro_long_log2 2024-04-25T09:41:09.680 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_trim_max_entries 2024-04-25T09:41:09.680 INFO:tasks.workunit.client.0.smithi134.stderr:TEST_trim_max_entries_with_dups' 2024-04-25T09:41:09.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:41:09.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:41:09.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:41:09.681 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:41:09.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:41:09.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:41:09.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:41:09.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:41:09.683 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:41:09.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:41:09.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:41:09.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:41:09.684 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:41:09.684 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:41:09.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:41:09.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:41:09.686 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:41:09.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:09.687 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:41:09.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494734 2024-04-25T09:41:09.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:41:09.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:41:09.688 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:41:09.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:41:09.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:09.689 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:41:09.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.494734 2024-04-25T09:41:09.690 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:41:09.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-04-25T09:41:09.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-04-25T09:41:09.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:41:09.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log1 td/repro-long-log 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:97: TEST_repro_long_log1: local dir=td/repro-long-log 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:99: TEST_repro_long_log1: setup_log_test td/repro-long-log 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:41:09.693 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:41:09.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:41:09.694 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:41:10.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:41:10.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:41:10.176 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:41:10.177 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:41:10.177 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:10.177 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:41:10.177 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:41:10.177 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:41:10.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:41:10.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:41:10.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:41:10.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:41:10.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:41:10.390 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:41:10.391 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:41:10.392 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:41:10.392 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:41:10.392 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:41:10.392 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:10.392 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:41:10.392 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494734/ceph-mon.a.asok 2024-04-25T09:41:10.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:41:10.392 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494734/ceph-mon.a.asok config get fsid 2024-04-25T09:41:10.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:41:10.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:41:10.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:41:10.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:41:10.508 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:41:10.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:41:10.508 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:41:10.509 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:41:10.509 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:41:10.509 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:10.510 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:41:10.510 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494734/ceph-mon.a.asok 2024-04-25T09:41:10.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:41:10.510 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494734/ceph-mon.a.asok config get mon_host 2024-04-25T09:41:10.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:41:10.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:41:10.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:41:10.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:41:10.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:41:10.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:41:10.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:41:11.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:41:11.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:41:11.544 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:41:11.545 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:41:11.545 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:11.545 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:41:11.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:41:11.545 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:41:11.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:41:11.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:41:11.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:41:11.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:41:11.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:41:11.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:41:11.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:11.580 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:41:11.581 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:41:11.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=03bba2fa-b037-4ca3-b085-177ddb001a82 2024-04-25T09:41:11.582 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 03bba2fa-b037-4ca3-b085-177ddb001a82 2024-04-25T09:41:11.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 03bba2fa-b037-4ca3-b085-177ddb001a82' 2024-04-25T09:41:11.583 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:41:11.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA3JSpmp10dJBAAL6pT5Bw3+UH1zWZtQjXdiA== 2024-04-25T09:41:11.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA3JSpmp10dJBAAL6pT5Bw3+UH1zWZtQjXdiA=="}' 2024-04-25T09:41:11.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 03bba2fa-b037-4ca3-b085-177ddb001a82 -i td/repro-long-log/0/new.json 2024-04-25T09:41:11.899 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:41:11.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:41:11.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3JSpmp10dJBAAL6pT5Bw3+UH1zWZtQjXdiA== --osd-uuid 03bba2fa-b037-4ca3-b085-177ddb001a82 2024-04-25T09:41:11.942 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:11.946+0000 7fbd6343e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:11.977 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:11.982+0000 7fbd6343e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:11.978 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:11.982+0000 7fbd6343e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:11.978 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:11.982+0000 7fbd6343e7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:41:11.979 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:11.982+0000 7fbd6343e7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:41:11.979 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:11.982+0000 7fbd6343e7c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:41:12.016 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:12.022+0000 7fbd6343e7c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:41:18.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:41:18.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:41:18.298 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:41:18.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:41:18.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:41:18.858 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:41:18.858 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:41:18.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:41:18.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:41:18.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:41:18.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:41:18.888 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:18.890+0000 7f94f44667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:18.890 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:18.894+0000 7f94f44667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:18.891 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:18.894+0000 7f94f44667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:19.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:41:19.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:41:19.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:41:19.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:41:19.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:41:19.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:19.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:41:19.321 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:41:19.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:19.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:19.453 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:19.458+0000 7f94f44667c0 -1 Falling back to public interface 2024-04-25T09:41:19.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:20.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:20.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:20.762 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:41:20.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:41:20.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:20.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:21.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:22.148 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:22.154+0000 7f94f44667c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:41:22.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:22.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:22.166 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:41:22.167 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:41:22.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:22.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:22.675 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:23.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:23.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:23.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:41:23.678 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:41:23.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:23.678 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:24.204 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:25.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:25.205 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:25.206 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:41:25.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:41:25.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:25.206 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:25.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:26.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:26.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:26.618 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:41:26.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:41:26.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:26.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:27.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:28.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:28.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:28.191 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:41:28.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:41:28.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:28.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:28.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:29.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:29.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:29.612 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:41:29.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:41:29.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:29.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:30.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:31.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:31.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:31.228 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:41:31.228 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:41:31.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:31.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:31.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:32.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:32.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:32.666 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:41:32.667 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:41:32.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:32.668 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:33.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:34.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:34.175 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:34.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:41:34.176 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:41:34.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:34.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:34.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:35.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:35.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:35.629 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:41:35.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:41:35.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:35.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:36.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:37.515 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:37.516 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:41:37.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:37.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:41:37.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:37.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:38.307 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:39.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:39.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:39.309 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:41:39.309 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:41:39.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:39.310 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:40.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:41.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:41.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:41.411 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:41:41.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:41:41.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:41.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:42.621 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:43.626 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:41:43.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:43.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:43.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:41:43.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:43.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:44.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:45.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:45.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:45.838 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:41:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:41:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:45.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:46.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:47.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:47.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:47.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:41:47.613 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:41:47.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:47.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:41:48.021 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1446937507,v1:127.0.0.1:6803/1446937507] [v2:127.0.0.1:6804/1446937507,v1:127.0.0.1:6805/1446937507] exists,up 03bba2fa-b037-4ca3-b085-177ddb001a82 2024-04-25T09:41:48.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:41:48.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:41:48.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:41:48.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:41:48.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:41:48.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:41:48.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:41:48.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:41:48.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:41:48.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:41:48.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:41:48.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:41:48.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:41:48.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:41:48.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8c682b3-d221-4d1a-91ab-08fdd351436c 2024-04-25T09:41:48.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 c8c682b3-d221-4d1a-91ab-08fdd351436c' 2024-04-25T09:41:48.028 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 c8c682b3-d221-4d1a-91ab-08fdd351436c 2024-04-25T09:41:48.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:41:48.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBcJSpmwFJNAxAAh/FknUesqwYNzs5JHoUJAw== 2024-04-25T09:41:48.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBcJSpmwFJNAxAAh/FknUesqwYNzs5JHoUJAw=="}' 2024-04-25T09:41:48.047 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8c682b3-d221-4d1a-91ab-08fdd351436c -i td/repro-long-log/1/new.json 2024-04-25T09:41:48.595 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:41:48.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:41:48.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBcJSpmwFJNAxAAh/FknUesqwYNzs5JHoUJAw== --osd-uuid c8c682b3-d221-4d1a-91ab-08fdd351436c 2024-04-25T09:41:48.639 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:48.645+0000 7f7dabd837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:48.680 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:48.685+0000 7f7dabd837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:48.681 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:48.689+0000 7f7dabd837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:48.682 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:48.689+0000 7f7dabd837c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:41:48.682 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:48.689+0000 7f7dabd837c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:41:48.682 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:48.689+0000 7f7dabd837c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:41:48.719 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:48.725+0000 7f7dabd837c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:41:55.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:41:55.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:41:55.212 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:41:55.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:41:55.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:41:55.771 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:41:55.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:41:55.772 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:41:55.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:41:55.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:41:55.773 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:41:55.800 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:55.805+0000 7f883b3767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:55.802 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:55.809+0000 7f883b3767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:55.803 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:55.809+0000 7f883b3767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:41:56.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:41:56.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:41:56.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:41:56.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:41:56.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:41:56.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:56.241 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:41:56.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:41:56.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:56.242 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:56.362 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:56.369+0000 7f883b3767c0 -1 Falling back to public interface 2024-04-25T09:41:56.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:57.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:57.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:57.653 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:41:57.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:41:57.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:57.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:58.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:41:59.050 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:41:59.057+0000 7f883b3767c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:41:59.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:41:59.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:41:59.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:41:59.058 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:41:59.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:41:59.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:41:59.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:00.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:00.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:00.617 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:42:00.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:42:00.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:00.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:01.034 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:02.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:02.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:02.036 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:42:02.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:42:02.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:02.037 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:02.504 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:03.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:03.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:03.506 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:42:03.506 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:42:03.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:03.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:04.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:05.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:05.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:05.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:42:05.096 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:42:05.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:05.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:05.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:06.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:06.541 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:06.542 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:42:06.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:42:06.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:06.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:07.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:08.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:08.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:08.059 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:42:08.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:42:08.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:08.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:08.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:09.525 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:09.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:09.526 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:42:09.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:42:09.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:09.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:10.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:11.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:11.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:11.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:42:11.051 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:42:11.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:11.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:11.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:12.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:12.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:12.549 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:42:12.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:42:12.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:12.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:13.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:14.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:14.633 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:42:14.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:14.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:42:14.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:14.633 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:15.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:16.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:16.698 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:16.698 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:42:16.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:42:16.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:16.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:17.626 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:18.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:18.627 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:18.628 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:42:18.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:42:18.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:18.628 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:19.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:20.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:20.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:20.784 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:42:20.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:42:20.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:20.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:21.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:22.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:22.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:22.266 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:42:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:42:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:22.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:22.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:23.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:23.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:23.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:42:23.920 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:42:23.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:23.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:42:24.675 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2408920929,v1:127.0.0.1:6811/2408920929] [v2:127.0.0.1:6812/2408920929,v1:127.0.0.1:6813/2408920929] exists,up c8c682b3-d221-4d1a-91ab-08fdd351436c 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:42:24.679 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:42:24.680 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:42:24.681 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:42:24.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:42:24.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fa9d12e0-6c33-45ec-afb5-b0e7f74e1556 2024-04-25T09:42:24.682 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 fa9d12e0-6c33-45ec-afb5-b0e7f74e1556' 2024-04-25T09:42:24.682 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 fa9d12e0-6c33-45ec-afb5-b0e7f74e1556 2024-04-25T09:42:24.682 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:42:24.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCAJSpmTK9aKhAA5L7T6Vl849CAKZ09lEFnMg== 2024-04-25T09:42:24.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCAJSpmTK9aKhAA5L7T6Vl849CAKZ09lEFnMg=="}' 2024-04-25T09:42:24.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fa9d12e0-6c33-45ec-afb5-b0e7f74e1556 -i td/repro-long-log/2/new.json 2024-04-25T09:42:25.206 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:42:25.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:42:25.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCAJSpmTK9aKhAA5L7T6Vl849CAKZ09lEFnMg== --osd-uuid fa9d12e0-6c33-45ec-afb5-b0e7f74e1556 2024-04-25T09:42:25.255 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:25.261+0000 7f95955f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:42:25.268 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:25.273+0000 7f95955f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:42:25.269 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:25.277+0000 7f95955f07c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:42:25.270 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:25.277+0000 7f95955f07c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:42:25.270 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:25.277+0000 7f95955f07c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:42:25.270 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:25.277+0000 7f95955f07c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:42:25.308 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:25.313+0000 7f95955f07c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:42:32.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:42:32.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:42:32.224 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:42:32.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:42:32.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:42:32.789 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:42:32.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:42:32.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:42:32.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:42:32.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:42:32.790 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:42:32.818 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:32.824+0000 7f87e464f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:42:32.833 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:32.840+0000 7f87e464f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:42:32.834 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:32.840+0000 7f87e464f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:42:33.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:42:33.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:42:33.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:42:33.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:42:33.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:42:33.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:33.186 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:42:33.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:42:33.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:33.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:33.391 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:33.396+0000 7f87e464f7c0 -1 Falling back to public interface 2024-04-25T09:42:33.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:34.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:34.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:34.657 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:42:34.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:42:34.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:34.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:35.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:36.061 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:42:36.064+0000 7f87e464f7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:42:36.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:36.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:36.079 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:42:36.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:42:36.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:36.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:36.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:37.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:37.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:37.582 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:42:37.583 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:42:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:37.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:38.158 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:39.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:39.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:39.160 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:42:39.160 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:42:39.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:39.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:39.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:40.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:40.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:40.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:42:40.709 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:42:40.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:40.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:41.398 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:42.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:42.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:42.400 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:42:42.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:42:42.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:42.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:42.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:43.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:43.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:43.960 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:42:43.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:42:43.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:43.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:44.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:45.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:45.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:45.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:42:45.531 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:42:45.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:45.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:45.947 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:46.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:46.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:46.949 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:42:46.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:42:46.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:46.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:47.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:48.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:48.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:48.423 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:42:48.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:42:48.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:48.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:48.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:49.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:49.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:49.975 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:42:49.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:42:49.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:49.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:50.397 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:51.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:51.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:51.399 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:42:51.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:42:51.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:51.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:52.098 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:53.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:53.100 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:53.101 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:42:53.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:42:53.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:53.101 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:53.781 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:54.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:54.783 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:42:54.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:54.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:42:54.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:54.784 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:55.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:56.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:56.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:56.421 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:42:56.421 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:42:56.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:56.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:57.281 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:42:58.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:42:58.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:42:58.283 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:42:58.283 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:42:58.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:42:58.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:42:59.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:00.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:00.404 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:00.405 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:43:00.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:43:00.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:00.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:43:01.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:02.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:02.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:02.062 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:43:02.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:43:02.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:02.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:43:02.535 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3727478836,v1:127.0.0.1:6819/3727478836] [v2:127.0.0.1:6820/3727478836,v1:127.0.0.1:6821/3727478836] exists,up fa9d12e0-6c33-45ec-afb5-b0e7f74e1556 2024-04-25T09:43:02.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:43:02.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:43:02.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:43:02.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:43:03.034 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T09:43:03.051 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:43:03.051 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:43:03.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:43:03.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:43:03.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:43:03.724 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:43:03.751 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:43:03.780 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:43:03.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:43:03.949 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:43:03.977 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:43:04.007 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:43:04.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:43:04.161 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:43:04.187 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:43:04.217 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:43:04.230 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:43:04.463 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:43:04.490 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:43:04.524 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:43:04.539 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:43:04.679 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:43:04.708 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:43:04.735 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:43:04.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:43:04.753 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:43:04.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:04.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:05.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:05.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:05.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:05.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:05.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:05.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:05.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:05.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:05.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:05.655 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:05.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:05.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:05.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:05.890 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:06.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:06.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:06.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:06.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:06.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:06.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:06.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:06.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:06.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:06.560 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:06.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:06.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:06.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:06.912 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:07.045 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:07.046 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:07.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:07.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:07.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:07.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:07.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:07.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:07.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:07.489 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:07.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:43:07.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:43:07.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:43:07.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:43:07.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:43:07.742 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836501 2024-04-25T09:43:07.765 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672975 2024-04-25T09:43:07.789 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509447 2024-04-25T09:43:07.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:43:10.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:10.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:43:10.973 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:43:10.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:10.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:43:10.973 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:43:11.124 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:43:11.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:11.125 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:43:11.277 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:43:11.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:11.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:43:11.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:43:11.417 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:43:11.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:43:11.584 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:43:11.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:11.585 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:11.689 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:11.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:11.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:11.823 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:11.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:11.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:11.940 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:11.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:11.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:12.057 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:12.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:12.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:12.199 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:12.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:12.203 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:12.478 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:12.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:12.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:12.584 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:12.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:12.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:12.751 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:12.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:12.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:12.910 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:12.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:12.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:13.053 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:13.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:13.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:13.170 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:13.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:13.174 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:13.296 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:13.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:13.299 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:13.438 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:13.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:13.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:13.589 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:13.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:13.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:13.706 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:13.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:13.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:13.823 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:13.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:13.827 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:13.974 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:13.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:13.977 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:14.091 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:14.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:14.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:14.250 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:14.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:43:14.253 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:43:14.434 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:43:14.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:43:14.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:43:14.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:43:14.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:43:14.438 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:43:14.586 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836504 2024-04-25T09:43:14.606 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672978 2024-04-25T09:43:14.631 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509450 2024-04-25T09:43:14.646 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:43:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:17.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:43:17.813 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T09:43:17.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:17.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:43:17.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:43:17.970 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T09:43:17.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:43:17.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:18.134 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:43:18.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:18.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:43:18.135 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:43:18.289 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:43:18.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:101: TEST_repro_long_log1: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:43:18.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:102: TEST_repro_long_log1: test_log_size 1.0 22 2024-04-25T09:43:18.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:43:18.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=22 2024-04-25T09:43:18.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:43:18.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:43:18.553 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836507 2024-04-25T09:43:18.577 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672981 2024-04-25T09:43:18.602 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509453 2024-04-25T09:43:18.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:43:21.619 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:21.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:43:21.786 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T09:43:21.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:21.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:43:21.788 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 22 2024-04-25T09:43:21.948 INFO:tasks.workunit.client.0.smithi134.stdout:22 2024-04-25T09:43:21.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:21.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:43:22.104 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:43:22.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:43:22.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:43:22.105 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:43:22.256 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:43:22.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:43:22.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:43:22.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:43:22.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:43:22.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:43:22.258 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:43:22.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:43:22.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:43:22.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:43:22.568 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:43:22.568 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:43:22.569 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:43:22.569 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:43:22.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:43:22.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:43:22.571 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:43:22.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:43:22.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:43:22.573 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:43:22.573 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:43:22.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:43:22.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:43:22.575 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:43:22.596 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:43:22.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:22.597 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:43:22.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494734 2024-04-25T09:43:22.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:43:22.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:43:22.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:43:22.601 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:43:22.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:43:22.602 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:43:22.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:43:22.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:43:22.604 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:43:22.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:43:22.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:43:22.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:43:22.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:43:22.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:43:22.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:43:22.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:43:22.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:43:22.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:22.609 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:43:22.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494734 2024-04-25T09:43:22.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:43:22.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:43:22.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:43:22.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:43:22.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:22.612 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:43:22.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.494734 2024-04-25T09:43:22.613 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:43:22.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:43:22.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:43:22.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:43:22.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_repro_long_log2 td/repro-long-log 2024-04-25T09:43:22.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:107: TEST_repro_long_log2: local dir=td/repro-long-log 2024-04-25T09:43:22.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:109: TEST_repro_long_log2: setup_log_test td/repro-long-log 2024-04-25T09:43:22.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:43:22.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:43:22.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:43:22.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:43:22.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:43:22.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:43:22.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:43:22.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:43:22.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:43:23.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:43:23.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:43:23.082 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:43:23.082 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:43:23.082 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:23.083 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:43:23.083 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:43:23.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:43:23.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:43:23.294 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:43:23.294 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:43:23.294 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:43:23.295 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:43:23.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:43:23.296 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:43:23.296 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:43:23.296 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:43:23.296 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:43:23.296 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:23.296 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:43:23.296 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494734/ceph-mon.a.asok 2024-04-25T09:43:23.296 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:43:23.297 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494734/ceph-mon.a.asok config get fsid 2024-04-25T09:43:23.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:43:23.415 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:43:23.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:43:23.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:43:23.416 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:43:23.417 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:43:23.417 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:43:23.417 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:43:23.417 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:43:23.417 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:23.417 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:43:23.417 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494734/ceph-mon.a.asok 2024-04-25T09:43:23.417 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:43:23.418 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494734/ceph-mon.a.asok config get mon_host 2024-04-25T09:43:23.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:43:23.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:43:23.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:43:23.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:43:23.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:43:23.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:43:23.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:43:24.441 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:43:24.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:43:24.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:43:24.442 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:43:24.442 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:24.442 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:43:24.442 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:43:24.443 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:43:24.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:43:24.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:43:24.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:43:24.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:43:24.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:43:24.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:43:24.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:43:24.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:43:24.476 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:43:24.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:43:24.478 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:43:24.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e7180605-592f-434c-b358-a1c5ebcea121 2024-04-25T09:43:24.479 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 e7180605-592f-434c-b358-a1c5ebcea121 2024-04-25T09:43:24.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e7180605-592f-434c-b358-a1c5ebcea121' 2024-04-25T09:43:24.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:43:24.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC8JSpmlcfTHRAAmstR2YM3rPxjy7Ku0JA6Eg== 2024-04-25T09:43:24.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC8JSpmlcfTHRAAmstR2YM3rPxjy7Ku0JA6Eg=="}' 2024-04-25T09:43:24.494 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e7180605-592f-434c-b358-a1c5ebcea121 -i td/repro-long-log/0/new.json 2024-04-25T09:43:24.779 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:43:24.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:43:24.795 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC8JSpmlcfTHRAAmstR2YM3rPxjy7Ku0JA6Eg== --osd-uuid e7180605-592f-434c-b358-a1c5ebcea121 2024-04-25T09:43:24.821 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:24.823+0000 7fd8abac27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:43:24.842 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:24.843+0000 7fd8abac27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:43:24.842 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:24.843+0000 7fd8abac27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:43:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:24.843+0000 7fd8abac27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:43:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:24.843+0000 7fd8abac27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:43:24.843 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:24.843+0000 7fd8abac27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:43:24.880 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:24.883+0000 7fd8abac27c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:43:31.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:43:31.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:43:31.230 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:43:31.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:43:31.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:43:31.782 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:43:31.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:43:31.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:43:31.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:43:31.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:43:31.783 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:43:31.811 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:31.815+0000 7fa17b2627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:43:31.813 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:31.815+0000 7fa17b2627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:43:31.814 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:31.819+0000 7fa17b2627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:43:32.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:43:32.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:43:32.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:43:32.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:43:32.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:43:32.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:32.172 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:43:32.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:43:32.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:32.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:32.379 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:32.383+0000 7fa17b2627c0 -1 Falling back to public interface 2024-04-25T09:43:32.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:33.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:33.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:33.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:43:33.602 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:43:33.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:33.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:34.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:34.978 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:43:34.979+0000 7fa17b2627c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:43:35.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:35.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:35.022 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:43:35.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:43:35.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:35.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:35.701 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:36.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:36.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:36.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:43:36.703 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:43:36.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:36.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:37.154 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:38.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:38.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:38.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:43:38.156 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:43:38.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:38.157 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:38.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:39.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:39.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:39.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:43:39.598 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:43:39.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:39.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:40.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:41.171 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:41.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:41.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:43:41.172 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:43:41.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:41.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:41.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:42.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:42.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:42.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:43:42.637 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:43:42.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:42.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:43.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:44.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:44.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:44.147 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:43:44.147 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:43:44.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:44.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:44.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:45.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:45.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:45.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:43:45.598 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:43:45.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:45.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:46.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:47.186 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:47.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:47.187 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:43:47.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:43:47.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:47.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:47.593 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:48.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:48.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:48.595 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:43:48.596 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:43:48.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:48.596 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:49.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:50.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:50.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:50.583 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:43:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:43:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:50.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:51.420 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:52.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:52.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:52.422 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:43:52.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:43:52.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:52.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:53.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:54.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:54.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:54.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:43:54.327 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:43:54.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:54.327 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:55.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:56.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:56.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:56.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:43:56.152 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:43:56.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:56.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:56.911 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:57.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:57.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:57.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:43:57.914 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:43:57.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:57.915 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:43:58.466 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:43:59.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:43:59.470 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:43:59.471 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:43:59.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:43:59.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:43:59.471 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:44:00.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:01.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:01.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:01.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:44:01.271 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:44:01.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:01.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:44:01.702 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1619375581,v1:127.0.0.1:6803/1619375581] [v2:127.0.0.1:6804/1619375581,v1:127.0.0.1:6805/1619375581] exists,up e7180605-592f-434c-b358-a1c5ebcea121 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:44:01.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:44:01.708 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:44:01.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:44:01.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:44:01.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:44:01.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2289b93d-6bfd-4a85-bc66-adb6716e53b8 2024-04-25T09:44:01.710 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2289b93d-6bfd-4a85-bc66-adb6716e53b8' 2024-04-25T09:44:01.710 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 2289b93d-6bfd-4a85-bc66-adb6716e53b8 2024-04-25T09:44:01.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:44:01.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDhJSpmaYKtKxAApZlqopJJgAySVSwCnShNOw== 2024-04-25T09:44:01.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDhJSpmaYKtKxAApZlqopJJgAySVSwCnShNOw=="}' 2024-04-25T09:44:01.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2289b93d-6bfd-4a85-bc66-adb6716e53b8 -i td/repro-long-log/1/new.json 2024-04-25T09:44:02.203 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:44:02.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:44:02.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDhJSpmaYKtKxAApZlqopJJgAySVSwCnShNOw== --osd-uuid 2289b93d-6bfd-4a85-bc66-adb6716e53b8 2024-04-25T09:44:02.252 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:02.254+0000 7f2bd2ab97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:02.256 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:02.258+0000 7f2bd2ab97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:02.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:02.258+0000 7f2bd2ab97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:02.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:02.258+0000 7f2bd2ab97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:44:02.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:02.258+0000 7f2bd2ab97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:44:02.258 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:02.258+0000 7f2bd2ab97c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:44:02.296 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:02.298+0000 7f2bd2ab97c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:44:09.070 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:44:09.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:44:09.072 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:44:09.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:44:09.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:44:09.628 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:44:09.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:44:09.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:44:09.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:44:09.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:44:09.629 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:44:09.657 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:09.658+0000 7fe1428537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:09.659 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:09.662+0000 7fe1428537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:09.660 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:09.662+0000 7fe1428537c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:10.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:44:10.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:44:10.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:44:10.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:44:10.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:44:10.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:10.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:44:10.024 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:44:10.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:10.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:10.228 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:10.230+0000 7fe1428537c0 -1 Falling back to public interface 2024-04-25T09:44:10.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:11.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:11.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:11.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:44:11.510 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:44:11.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:11.510 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:11.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:12.885 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:12.886+0000 7fe1428537c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:44:12.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:12.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:12.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:44:12.918 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:44:12.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:12.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:13.608 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:14.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:14.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:14.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:44:14.611 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:44:14.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:14.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:15.073 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:16.075 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:16.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:16.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:44:16.076 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:44:16.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:16.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:16.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:17.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:17.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:17.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:44:17.536 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:44:17.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:17.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:18.163 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:19.164 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:19.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:19.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:44:19.165 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:44:19.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:19.165 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:19.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:20.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:20.616 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:20.617 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:44:20.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:44:20.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:20.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:21.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:22.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:22.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:22.071 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:44:22.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:44:22.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:22.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:22.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:23.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:23.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:23.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:44:23.502 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:44:23.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:23.502 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:24.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:25.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:25.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:25.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:44:25.011 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:44:25.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:25.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:25.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:26.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:26.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:26.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:44:26.446 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:44:26.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:26.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:26.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:27.975 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:27.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:27.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:44:27.978 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:44:27.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:27.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:28.736 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:29.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:29.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:29.738 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:44:29.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:44:29.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:29.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:30.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:31.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:31.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:31.537 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:44:31.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:44:31.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:31.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:32.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:33.401 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:33.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:33.402 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:44:33.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:44:33.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:33.403 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:34.208 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:35.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:35.209 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:35.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:44:35.210 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:44:35.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:35.210 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:36.093 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:37.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:37.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:37.095 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:44:37.095 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:44:37.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:37.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:37.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:38.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:38.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:38.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:44:38.703 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:44:38.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:38.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:44:39.114 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3846031545,v1:127.0.0.1:6811/3846031545] [v2:127.0.0.1:6812/3846031545,v1:127.0.0.1:6813/3846031545] exists,up 2289b93d-6bfd-4a85-bc66-adb6716e53b8 2024-04-25T09:44:39.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:44:39.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:44:39.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:44:39.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:44:39.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:44:39.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:44:39.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:44:39.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:44:39.120 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:44:39.120 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:44:39.120 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:44:39.121 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:44:39.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:44:39.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:44:39.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:44:39.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:44:39.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:44:39.122 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:44:39.122 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:44:39.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=62e19fa9-0316-4df5-9fbe-2054a26ff50e 2024-04-25T09:44:39.123 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 62e19fa9-0316-4df5-9fbe-2054a26ff50e' 2024-04-25T09:44:39.123 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 62e19fa9-0316-4df5-9fbe-2054a26ff50e 2024-04-25T09:44:39.123 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:44:39.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHJipm9xacCBAAoPsJz5ZIlN8zPvYq7/GvNA== 2024-04-25T09:44:39.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHJipm9xacCBAAoPsJz5ZIlN8zPvYq7/GvNA=="}' 2024-04-25T09:44:39.139 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 62e19fa9-0316-4df5-9fbe-2054a26ff50e -i td/repro-long-log/2/new.json 2024-04-25T09:44:39.620 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:44:39.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:44:39.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAHJipm9xacCBAAoPsJz5ZIlN8zPvYq7/GvNA== --osd-uuid 62e19fa9-0316-4df5-9fbe-2054a26ff50e 2024-04-25T09:44:39.667 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:39.673+0000 7f4a7ced77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:39.670 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:39.673+0000 7f4a7ced77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:39.671 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:39.677+0000 7f4a7ced77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:39.671 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:39.677+0000 7f4a7ced77c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:44:39.672 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:39.677+0000 7f4a7ced77c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:44:39.672 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:39.677+0000 7f4a7ced77c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:44:39.753 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:39.757+0000 7f4a7ced77c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:44:46.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:44:46.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:44:46.606 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:44:46.606 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:44:46.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:44:47.186 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:44:47.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:44:47.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:44:47.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:44:47.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:44:47.187 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:44:47.216 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:47.221+0000 7f6be47e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:47.217 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:47.221+0000 7f6be47e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:47.219 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:47.221+0000 7f6be47e67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:44:47.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:44:47.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:44:47.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:44:47.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:44:47.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:44:47.664 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:47.665 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:44:47.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:44:47.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:47.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:44:47.800 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:47.805+0000 7f6be47e67c0 -1 Falling back to public interface 2024-04-25T09:44:48.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:49.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:49.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:49.083 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:44:49.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:44:49.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:49.083 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:44:49.488 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:50.476 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:44:50.481+0000 7f6be47e67c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:44:50.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:50.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:50.490 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:44:50.490 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:44:50.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:44:50.491 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:51.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:52.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:52.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:52.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:44:52.329 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:44:52.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:52.329 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:44:52.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:53.804 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:53.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:53.805 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:44:53.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:44:53.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:53.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:44:54.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:55.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:55.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:55.268 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:44:55.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:44:55.268 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:55.269 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:44:55.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:56.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:56.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:56.685 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:44:56.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:44:56.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:56.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:44:57.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:58.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:58.111 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:44:58.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:44:58.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:58.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:44:58.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:44:59.520 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:44:59.521 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:44:59.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:44:59.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:44:59.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:44:59.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:00.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:01.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:01.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:01.013 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:45:01.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:45:01.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:01.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:01.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:02.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:02.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:02.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:45:02.617 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:45:02.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:02.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:03.050 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:04.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:04.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:04.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:45:04.052 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:45:04.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:04.053 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:04.528 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:05.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:05.529 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:05.530 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:45:05.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:45:05.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:05.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:06.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:07.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:07.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:07.652 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:45:07.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:45:07.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:07.652 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:08.276 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:09.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:09.278 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:09.278 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:45:09.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:45:09.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:09.279 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:10.423 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:11.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:11.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:11.424 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:45:11.425 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:45:11.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:11.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:11.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:12.988 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:12.989 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:45:12.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:12.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:45:12.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:12.989 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:13.531 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:14.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:14.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:14.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:45:14.533 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:45:14.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:14.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:15.287 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:16.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:16.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:16.289 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:45:16.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:45:16.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:16.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:16.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:17.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:17.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:18.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-04-25T09:45:18.000 INFO:tasks.workunit.client.0.smithi134.stdout:19 2024-04-25T09:45:18.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:18.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:45:18.407 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2994584089,v1:127.0.0.1:6819/2994584089] [v2:127.0.0.1:6820/2994584089,v1:127.0.0.1:6821/2994584089] exists,up 62e19fa9-0316-4df5-9fbe-2054a26ff50e 2024-04-25T09:45:18.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:45:18.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:45:18.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:45:18.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:45:18.964 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T09:45:18.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:45:18.981 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:45:19.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:45:19.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:45:19.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:45:19.580 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:45:19.610 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:45:19.637 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:45:19.650 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:45:19.788 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:45:19.815 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:45:19.842 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:45:19.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:45:20.011 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:45:20.037 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:45:20.063 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:45:20.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:45:20.221 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:45:20.249 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:45:20.281 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:45:20.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:45:20.530 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:45:20.560 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:45:20.589 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:45:20.605 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:45:20.606 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:45:20.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:20.607 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:21.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:21.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:21.414 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:21.602 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:21.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:21.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:21.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:21.837 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:21.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:21.969 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:22.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:22.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:22.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:22.238 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:22.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:22.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:22.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:22.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:22.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:22.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:22.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:22.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:22.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:22.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:22.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:22.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:23.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:23.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:23.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:23.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:23.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:23.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:23.518 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:23.685 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:23.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:45:23.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:45:23.802 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:45:23.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:45:23.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:45:23.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:45:23.947 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836512 2024-04-25T09:45:23.971 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672977 2024-04-25T09:45:23.995 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509449 2024-04-25T09:45:24.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:45:27.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:27.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:45:27.178 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:45:27.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:27.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:45:27.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:45:27.330 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:45:27.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:27.330 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:45:27.485 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:45:27.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:27.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:45:27.486 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:45:27.631 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:45:27.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:45:27.738 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:45:27.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:27.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:27.877 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:27.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:27.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:28.019 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:28.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:28.022 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:28.128 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:28.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:28.131 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:28.236 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:28.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:28.240 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:28.345 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:28.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:28.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:28.454 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:28.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:28.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:28.563 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:28.566 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:28.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:28.680 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:28.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:28.683 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:28.813 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:28.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:28.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:28.975 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:28.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:28.979 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:29.103 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:29.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:29.106 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:29.209 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:29.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:29.213 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:29.318 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:29.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:29.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:29.426 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:29.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:29.430 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:29.594 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:29.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:29.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:29.786 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:29.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:29.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:29.895 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:29.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:29.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:30.004 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:30.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:30.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:30.121 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:30.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:45:30.124 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:45:30.230 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:45:30.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:45:30.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:45:30.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:45:30.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:45:30.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:45:30.379 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836515 2024-04-25T09:45:30.404 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672980 2024-04-25T09:45:30.428 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509452 2024-04-25T09:45:30.443 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:45:33.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:33.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:45:33.631 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T09:45:33.631 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:33.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:45:33.632 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:45:33.785 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T09:45:33.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:33.785 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:45:33.927 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:45:33.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:33.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:45:33.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:45:34.071 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:45:34.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: ceph pg 1.0 query 2024-04-25T09:45:34.072 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: jq .info.stats.up_primary 2024-04-25T09:45:34.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:110: TEST_repro_long_log2: local PRIMARY=1 2024-04-25T09:45:34.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:111: TEST_repro_long_log2: kill_daemons td/repro-long-log TERM osd.1 2024-04-25T09:45:34.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:45:34.222 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:45:34.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:45:34.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:45:34.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:45:34.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:45:34.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: CEPH_ARGS='--osd-max-pg-log-entries=2 --osd-pg-log-dups-tracked=3 --no-mon-config' 2024-04-25T09:45:34.530 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:112: TEST_repro_long_log2: ceph-objectstore-tool --data-path td/repro-long-log/1 --pgid 1.0 --op trim-pg-log 2024-04-25T09:45:36.099 INFO:tasks.workunit.client.0.smithi134.stderr:Log bounds are: (19'20,19'41] 2024-04-25T09:45:36.099 INFO:tasks.workunit.client.0.smithi134.stdout:Removing keys 0000000019.00000000000000000021 - 0000000019.00000000000000000039 2024-04-25T09:45:36.131 INFO:tasks.workunit.client.0.smithi134.stderr:Finished trimming, now compacting... 2024-04-25T09:45:37.185 INFO:tasks.workunit.client.0.smithi134.stdout:Finished trimming pg log 2024-04-25T09:45:37.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:113: TEST_repro_long_log2: activate_osd td/repro-long-log 1 2024-04-25T09:45:37.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/repro-long-log 2024-04-25T09:45:37.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-04-25T09:45:37.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-04-25T09:45:37.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-04-25T09:45:37.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:45:37.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:45:37.636 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:45:37.637 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:45:37.637 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:45:37.637 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-04-25T09:45:37.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:45:37.639 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:45:37.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-04-25T09:45:37.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:45:37.639 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/repro-long-log/1/whoami 2024-04-25T09:45:37.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-04-25T09:45:37.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-04-25T09:45:37.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-04-25T09:45:37.641 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-04-25T09:45:37.672 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:37.676+0000 7f2f0bb727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:45:37.674 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:37.676+0000 7f2f0bb727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:45:37.676 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:37.680+0000 7f2f0bb727c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:45:38.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-04-25T09:45:38.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:45:38.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:45:38.057 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:45:38.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:45:38.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:45:38.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:38.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:45:38.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:45:38.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:38.229 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:38.232+0000 7f2f0bb727c0 -1 Falling back to public interface 2024-04-25T09:45:38.475 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:39.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:39.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:39.477 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:45:39.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:45:39.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:39.478 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:45:39.900 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:45:40.710 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:40.712+0000 7f2f0bb727c0 -1 osd.1 19 log_to_monitors true 2024-04-25T09:45:40.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:45:40.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:45:40.902 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:45:40.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:45:40.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:45:40.903 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:45:41.407 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 23 up_thru 23 down_at 20 last_clean_interval [10,19) [v2:127.0.0.1:6810/2478317694,v1:127.0.0.1:6811/2478317694] [v2:127.0.0.1:6812/2478317694,v1:127.0.0.1:6813/2478317694] exists,up 2289b93d-6bfd-4a85-bc66-adb6716e53b8 2024-04-25T09:45:41.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:45:41.408 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:45:41.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:45:41.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:114: TEST_repro_long_log2: wait_for_clean 2024-04-25T09:45:41.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-04-25T09:45:41.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-04-25T09:45:41.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-04-25T09:45:41.409 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-04-25T09:45:41.409 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-04-25T09:45:41.409 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-04-25T09:45:41.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-04-25T09:45:41.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-04-25T09:45:41.410 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-04-25T09:45:41.507 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-04-25T09:45:41.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-04-25T09:45:41.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-04-25T09:45:41.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-04-25T09:45:41.508 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2213: flush_pg_stats: local timeout=300 2024-04-25T09:45:41.508 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ceph osd ls 2024-04-25T09:45:41.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: flush_pg_stats: ids='0 2024-04-25T09:45:41.957 INFO:tasks.workunit.client.0.smithi134.stderr:1 2024-04-25T09:45:41.957 INFO:tasks.workunit.client.0.smithi134.stderr:2' 2024-04-25T09:45:41.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2216: flush_pg_stats: seqs= 2024-04-25T09:45:41.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:45:41.958 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-04-25T09:45:42.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=21474836520 2024-04-25T09:45:42.113 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 21474836520 2024-04-25T09:45:42.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520' 2024-04-25T09:45:42.114 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:45:42.114 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-04-25T09:45:42.251 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=98784247811 2024-04-25T09:45:42.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 98784247811 2024-04-25T09:45:42.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-98784247811' 2024-04-25T09:45:42.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: flush_pg_stats: for osd in $ids 2024-04-25T09:45:42.252 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-04-25T09:45:42.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2218: flush_pg_stats: seq=64424509456 2024-04-25T09:45:42.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: flush_pg_stats: test -z 64424509456 2024-04-25T09:45:42.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: flush_pg_stats: seqs=' 0-21474836520 1-98784247811 2-64424509456' 2024-04-25T09:45:42.429 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:45:42.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 0-21474836520 2024-04-25T09:45:42.430 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:45:42.431 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=0 2024-04-25T09:45:42.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 0-21474836520 2024-04-25T09:45:42.431 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:45:42.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=21474836520 2024-04-25T09:45:42.433 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.0 seq 21474836520' 2024-04-25T09:45:42.433 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.0 seq 21474836520 2024-04-25T09:45:42.433 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:45:42.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836517 -lt 21474836520 2024-04-25T09:45:42.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: flush_pg_stats: sleep 1 2024-04-25T09:45:43.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: flush_pg_stats: '[' 300 -eq 0 ']' 2024-04-25T09:45:43.832 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 0 2024-04-25T09:45:44.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 21474836520 -lt 21474836520 2024-04-25T09:45:44.237 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:45:44.237 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 1-98784247811 2024-04-25T09:45:44.238 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:45:44.239 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=1 2024-04-25T09:45:44.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 1-98784247811 2024-04-25T09:45:44.240 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:45:44.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=98784247811 2024-04-25T09:45:44.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.1 seq 98784247811' 2024-04-25T09:45:44.241 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.1 seq 98784247811 2024-04-25T09:45:44.242 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 1 2024-04-25T09:45:44.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 98784247811 -lt 98784247811 2024-04-25T09:45:44.709 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: flush_pg_stats: for s in $seqs 2024-04-25T09:45:44.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: echo 2-64424509456 2024-04-25T09:45:44.710 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: cut -d - -f 1 2024-04-25T09:45:44.711 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: flush_pg_stats: osd=2 2024-04-25T09:45:44.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: echo 2-64424509456 2024-04-25T09:45:44.712 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: cut -d - -f 2 2024-04-25T09:45:44.713 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: flush_pg_stats: seq=64424509456 2024-04-25T09:45:44.713 INFO:tasks.workunit.client.0.smithi134.stdout:waiting osd.2 seq 64424509456 2024-04-25T09:45:44.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-04-25T09:45:44.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: ceph osd last-stat-seq 2 2024-04-25T09:45:45.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: flush_pg_stats: test 64424509457 -lt 64424509456 2024-04-25T09:45:45.170 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-04-25T09:45:45.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:45:45.171 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:45:45.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 1 == 0 2024-04-25T09:45:45.774 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-04-25T09:45:45.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-04-25T09:45:45.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-04-25T09:45:45.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-04-25T09:45:45.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-04-25T09:45:45.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-04-25T09:45:45.775 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-04-25T09:45:46.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=1 2024-04-25T09:45:46.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-04-25T09:45:46.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-04-25T09:45:46.200 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-04-25T09:45:46.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 1 = 1 2024-04-25T09:45:46.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-04-25T09:45:46.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-04-25T09:45:46.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:115: TEST_repro_long_log2: test_log_size 1.0 21 18 2024-04-25T09:45:46.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:45:46.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:45:46.716 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-04-25T09:45:46.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:45:46.841 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836523 2024-04-25T09:45:46.863 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 98784247814 2024-04-25T09:45:46.886 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509459 2024-04-25T09:45:46.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:45:49.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:49.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:45:50.063 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:45:50.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:50.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:45:50.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:45:50.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:50.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:45:50.369 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:45:50.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:45:50.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:45:50.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-04-25T09:45:50.515 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:45:50.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:45:50.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:45:50.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:45:50.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:45:50.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:45:50.516 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:45:50.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:45:50.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:45:50.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:45:50.823 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:45:50.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:45:50.824 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:45:50.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:45:50.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:45:50.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:45:50.826 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:45:50.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:45:50.828 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:45:50.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:45:50.829 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:45:50.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:45:50.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:45:50.831 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:45:50.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:45:50.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:45:50.853 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:45:50.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494734 2024-04-25T09:45:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:45:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:45:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:45:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:45:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:45:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:45:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:45:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:45:50.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:45:50.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:45:50.857 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:45:50.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:45:50.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:45:50.857 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:45:50.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:45:50.860 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:45:50.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:45:50.862 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:45:50.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:45:50.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:45:50.863 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:45:50.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:45:50.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:45:50.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:45:50.865 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:45:50.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:45:50.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:45:50.867 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:45:50.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:45:50.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:45:50.868 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:45:50.869 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494734 2024-04-25T09:45:50.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:45:50.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:45:50.870 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:45:50.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:45:50.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:45:50.871 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:45:50.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.494734 2024-04-25T09:45:50.874 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:45:50.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:45:50.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:45:50.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:45:50.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries td/repro-long-log 2024-04-25T09:45:50.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:120: TEST_trim_max_entries: local dir=td/repro-long-log 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:122: TEST_trim_max_entries: setup_log_test td/repro-long-log 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:45:50.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:45:51.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:45:51.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:45:51.353 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:45:51.354 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:45:51.354 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:45:51.354 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:45:51.354 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:45:51.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:45:51.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:45:51.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:45:51.640 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:45:51.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:45:51.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:45:51.641 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:45:51.641 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:45:51.641 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:45:51.641 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:45:51.642 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:45:51.642 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:45:51.642 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:45:51.642 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494734/ceph-mon.a.asok 2024-04-25T09:45:51.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:45:51.643 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494734/ceph-mon.a.asok config get fsid 2024-04-25T09:45:51.759 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:45:51.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:45:51.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:45:51.760 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:45:51.761 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:45:51.761 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:45:51.761 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:45:51.761 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:45:51.762 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:45:51.762 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:45:51.762 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:45:51.762 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494734/ceph-mon.a.asok 2024-04-25T09:45:51.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:45:51.762 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494734/ceph-mon.a.asok config get mon_host 2024-04-25T09:45:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:45:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:45:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:45:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:45:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:45:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:45:51.879 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:45:52.805 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:45:52.806 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:45:52.806 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:45:52.806 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:45:52.806 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:45:52.806 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:45:52.806 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:45:52.807 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:45:52.807 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:45:52.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:45:52.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:45:52.845 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:45:52.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:45:52.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d0859bfd-967a-4e29-99dc-09f992bfb153 2024-04-25T09:45:52.847 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 d0859bfd-967a-4e29-99dc-09f992bfb153 2024-04-25T09:45:52.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d0859bfd-967a-4e29-99dc-09f992bfb153' 2024-04-25T09:45:52.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:45:52.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBQJipmV9eOMxAAuyP5OXjCKqDpVSgSMh85AA== 2024-04-25T09:45:52.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBQJipmV9eOMxAAuyP5OXjCKqDpVSgSMh85AA=="}' 2024-04-25T09:45:52.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d0859bfd-967a-4e29-99dc-09f992bfb153 -i td/repro-long-log/0/new.json 2024-04-25T09:45:53.151 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:45:53.167 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:45:53.169 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBQJipmV9eOMxAAuyP5OXjCKqDpVSgSMh85AA== --osd-uuid d0859bfd-967a-4e29-99dc-09f992bfb153 2024-04-25T09:45:53.205 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:53.208+0000 7fb9fcf407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:45:53.220 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:53.220+0000 7fb9fcf407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:45:53.221 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:53.224+0000 7fb9fcf407c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:45:53.222 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:53.224+0000 7fb9fcf407c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:45:53.222 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:53.224+0000 7fb9fcf407c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:45:53.222 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:53.224+0000 7fb9fcf407c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:45:53.259 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:45:53.260+0000 7fb9fcf407c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:45:59.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:45:59.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:45:59.728 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:45:59.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:45:59.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:46:00.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:46:00.293 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:46:00.293 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:46:00.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:46:00.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:46:00.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:46:00.322 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:00.323+0000 7f81c54d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:46:00.324 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:00.323+0000 7f81c54d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:46:00.325 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:00.327+0000 7f81c54d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:46:00.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:46:00.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:46:00.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:46:00.749 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:46:00.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:46:00.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:46:00.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:00.749 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:46:00.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:00.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:00.902 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:00.903+0000 7f81c54d47c0 -1 Falling back to public interface 2024-04-25T09:46:01.141 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:02.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:02.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:02.143 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:46:02.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:46:02.143 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:02.144 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:02.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:03.536 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:03.539+0000 7f81c54d47c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:46:03.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:03.554 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:03.555 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:46:03.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:46:03.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:03.555 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:04.076 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:05.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:05.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:05.078 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:46:05.078 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:46:05.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:05.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:05.511 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:06.512 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:06.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:06.513 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:46:06.513 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:46:06.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:06.514 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:07.079 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:08.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:08.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:08.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:46:08.081 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:46:08.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:08.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:08.535 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:09.536 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:09.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:09.537 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:46:09.537 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:46:09.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:09.538 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:09.952 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:10.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:10.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:10.955 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:46:10.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:46:10.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:10.955 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:11.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:12.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:12.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:12.542 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:46:12.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:46:12.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:12.543 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:12.963 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:13.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:13.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:13.964 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:46:13.965 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:46:13.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:13.965 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:14.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:15.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:15.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:15.549 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:46:15.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:46:15.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:15.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:15.957 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:16.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:16.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:16.960 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:46:16.961 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:46:16.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:16.961 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:17.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:18.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:18.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:18.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:46:18.368 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:46:18.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:18.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:18.939 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:19.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:19.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:19.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:46:19.942 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:46:19.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:19.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:20.659 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:21.661 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:21.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:21.662 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:46:21.662 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:46:21.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:21.663 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:22.176 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:23.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:23.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:23.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:46:23.179 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:46:23.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:23.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:23.985 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:24.986 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:24.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:24.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:46:24.987 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:46:24.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:24.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:25.756 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:26.757 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:26.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:26.758 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:46:26.758 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:46:26.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:26.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:27.803 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:28.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:28.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:28.805 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:46:28.805 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:46:28.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:28.806 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:46:29.257 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2184448197,v1:127.0.0.1:6803/2184448197] [v2:127.0.0.1:6804/2184448197,v1:127.0.0.1:6805/2184448197] exists,up d0859bfd-967a-4e29-99dc-09f992bfb153 2024-04-25T09:46:29.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:46:29.260 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:46:29.261 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:46:29.263 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:46:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:46:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:46:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:46:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:46:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:46:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:46:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:46:29.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f7f6b1ae-4d2e-4d48-b85d-4ed001f0257d 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f7f6b1ae-4d2e-4d48-b85d-4ed001f0257d' 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 f7f6b1ae-4d2e-4d48-b85d-4ed001f0257d 2024-04-25T09:46:29.264 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:46:29.284 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQB1JipmXUQ0ERAAgvTLrkxIn+RcTZBTuv5K+g== 2024-04-25T09:46:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQB1JipmXUQ0ERAAgvTLrkxIn+RcTZBTuv5K+g=="}' 2024-04-25T09:46:29.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f7f6b1ae-4d2e-4d48-b85d-4ed001f0257d -i td/repro-long-log/1/new.json 2024-04-25T09:46:29.743 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:46:29.760 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:46:29.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB1JipmXUQ0ERAAgvTLrkxIn+RcTZBTuv5K+g== --osd-uuid f7f6b1ae-4d2e-4d48-b85d-4ed001f0257d 2024-04-25T09:46:29.790 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:29.791+0000 7fa07cebb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:46:29.819 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:29.819+0000 7fa07cebb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:46:29.821 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:29.823+0000 7fa07cebb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:46:29.821 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:29.823+0000 7fa07cebb7c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:46:29.821 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:29.823+0000 7fa07cebb7c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:46:29.821 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:29.823+0000 7fa07cebb7c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:46:29.858 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:29.859+0000 7fa07cebb7c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:46:36.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:46:36.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:46:36.612 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:46:36.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:46:36.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:46:37.144 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:46:37.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:46:37.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:46:37.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:46:37.145 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:46:37.146 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:46:37.169 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:37.171+0000 7fdf01d6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:46:37.171 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:37.171+0000 7fdf01d6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:46:37.172 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:37.171+0000 7fdf01d6c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:46:37.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:46:37.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:46:37.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:46:37.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:46:37.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:46:37.578 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:37.578 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:46:37.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:46:37.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:37.579 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:37.718 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:37.718+0000 7fdf01d6c7c0 -1 Falling back to public interface 2024-04-25T09:46:38.004 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:39.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:39.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:39.006 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:46:39.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:46:39.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:39.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:39.409 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:40.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:40.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:40.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:46:40.411 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:46:40.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:40.412 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:40.495 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:46:40.494+0000 7fdf01d6c7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:46:40.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:41.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:41.865 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:41.865 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:46:41.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:46:41.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:41.866 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:42.298 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:43.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:43.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:43.300 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:46:43.300 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:46:43.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:43.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:43.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:44.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:44.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:44.734 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:46:44.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:46:44.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:44.735 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:45.170 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:46.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:46.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:46.173 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:46:46.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:46:46.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:46.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:46.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:47.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:47.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:47.614 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:46:47.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:46:47.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:47.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:48.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:49.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:49.233 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:49.233 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:46:49.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:46:49.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:49.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:49.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:50.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:50.768 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:50.768 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:46:50.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:46:50.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:50.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:51.181 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:52.183 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:52.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:52.184 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:46:52.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:46:52.184 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:52.185 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:52.598 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:53.599 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:53.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:53.600 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:46:53.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:46:53.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:53.600 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:54.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:55.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:55.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:55.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:46:55.015 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:46:55.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:55.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:56.454 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:57.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:57.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:57.457 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:46:57.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:46:57.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:57.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:58.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:46:59.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:46:59.086 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:46:59.086 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:46:59.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:46:59.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:46:59.087 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:46:59.882 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:00.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:00.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:00.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:47:00.884 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:47:00.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:00.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:47:01.762 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:02.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:02.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:02.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:47:02.766 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:47:02.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:02.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:47:03.613 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:04.614 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:04.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:04.615 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:47:04.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:47:04.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:04.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:47:05.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:06.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:06.724 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:06.724 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:47:06.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:47:06.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:06.725 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:47:07.146 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/160301831,v1:127.0.0.1:6811/160301831] [v2:127.0.0.1:6812/160301831,v1:127.0.0.1:6813/160301831] exists,up f7f6b1ae-4d2e-4d48-b85d-4ed001f0257d 2024-04-25T09:47:07.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:47:07.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:47:07.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:47:07.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:47:07.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:47:07.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:47:07.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:47:07.149 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:47:07.150 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:47:07.151 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:47:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:47:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:47:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:47:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:47:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:47:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:47:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:47:07.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:47:07.153 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:47:07.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6404dbd9-dd28-4825-8d6e-108e6caeebe6 2024-04-25T09:47:07.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6404dbd9-dd28-4825-8d6e-108e6caeebe6' 2024-04-25T09:47:07.153 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 6404dbd9-dd28-4825-8d6e-108e6caeebe6 2024-04-25T09:47:07.153 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:47:07.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCbJipmvVOBChAAJ+4rW8KbOcf9k4FYX58DxA== 2024-04-25T09:47:07.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCbJipmvVOBChAAJ+4rW8KbOcf9k4FYX58DxA=="}' 2024-04-25T09:47:07.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6404dbd9-dd28-4825-8d6e-108e6caeebe6 -i td/repro-long-log/2/new.json 2024-04-25T09:47:07.761 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:47:07.778 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:47:07.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbJipmvVOBChAAJ+4rW8KbOcf9k4FYX58DxA== --osd-uuid 6404dbd9-dd28-4825-8d6e-108e6caeebe6 2024-04-25T09:47:07.807 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:07.806+0000 7f848e4347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:47:07.834 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:07.834+0000 7f848e4347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:47:07.835 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:07.834+0000 7f848e4347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:47:07.836 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:07.834+0000 7f848e4347c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:47:07.836 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:07.834+0000 7f848e4347c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:47:07.836 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:07.834+0000 7f848e4347c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:47:07.880 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:07.878+0000 7f848e4347c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:47:14.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:47:14.958 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:47:14.959 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:47:14.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:47:14.959 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:47:15.516 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:47:15.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:47:15.516 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:47:15.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:47:15.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:47:15.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:47:15.544 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:15.546+0000 7efd85d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:47:15.546 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:15.550+0000 7efd85d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:47:15.548 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:15.550+0000 7efd85d0a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:47:15.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:47:15.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:47:15.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:47:15.916 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:47:15.916 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:47:15.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:47:15.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:15.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:47:15.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:15.917 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:16.094 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:16.098+0000 7efd85d0a7c0 -1 Falling back to public interface 2024-04-25T09:47:16.352 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:17.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:17.354 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:17.354 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:47:17.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:47:17.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:17.355 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:17.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:18.740 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:47:18.742+0000 7efd85d0a7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:47:18.813 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:18.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:18.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:47:18.814 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:47:18.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:18.815 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:19.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:20.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:20.546 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:20.547 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:47:20.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:47:20.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:20.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:20.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:21.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:21.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:21.998 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:47:21.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:47:21.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:21.999 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:22.702 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:23.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:23.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:23.704 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:47:23.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:47:23.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:23.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:24.216 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:25.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:25.218 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:25.219 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:47:25.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:47:25.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:25.219 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:25.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:26.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:26.750 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:26.751 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:47:26.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:47:26.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:26.751 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:27.222 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:28.223 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:28.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:28.224 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:47:28.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:47:28.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:28.225 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:28.798 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:29.799 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:29.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:29.800 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:47:29.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:47:29.800 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:29.801 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:30.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:31.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:31.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:31.392 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:47:31.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:47:31.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:31.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:32.224 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:33.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:33.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:33.227 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:47:33.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:47:33.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:33.227 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:33.638 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:34.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:34.639 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:34.640 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:47:34.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:47:34.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:34.640 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:35.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:36.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:36.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:36.324 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:47:36.324 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:47:36.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:36.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:37.364 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:38.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:38.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:38.366 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:47:38.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:47:38.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:38.367 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:39.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:40.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:40.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:40.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:47:40.290 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:47:40.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:40.291 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:40.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:41.942 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:41.942 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:47:41.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:47:41.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:41.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:43.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:44.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:44.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:44.084 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:47:44.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:47:44.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:44.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:44.524 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:47:45.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:47:45.526 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:47:45.527 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:47:45.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:47:45.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:47:45.527 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:47:45.953 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2889001732,v1:127.0.0.1:6819/2889001732] [v2:127.0.0.1:6820/2889001732,v1:127.0.0.1:6821/2889001732] exists,up 6404dbd9-dd28-4825-8d6e-108e6caeebe6 2024-04-25T09:47:45.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:47:45.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:47:45.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:47:45.954 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:47:46.462 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T09:47:46.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:47:46.479 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:47:47.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:47:47.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:47:47.059 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:47:47.260 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:47:47.289 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:47:47.317 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:47:47.331 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:47:47.533 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:47:47.568 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:47:47.604 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:47:47.620 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:47:47.838 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:47:47.866 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:47:47.898 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:47:47.914 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:47:48.119 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:47:48.150 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:47:48.178 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:47:48.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:47:48.418 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:47:48.448 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:47:48.480 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:47:48.496 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:47:48.497 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:47:48.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:48.498 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:48.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:48.881 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:49.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:49.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:49.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:49.148 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:49.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:49.265 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:49.399 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:49.400 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:49.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:49.558 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:49.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:49.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:49.793 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:49.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:49.943 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:49.944 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:50.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:50.077 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:50.211 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:50.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:50.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:50.328 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:50.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:50.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:50.571 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:50.572 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:50.722 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:50.723 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:50.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:50.889 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:50.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:50.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:51.132 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:51.133 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:51.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:51.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:47:51.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:47:51.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:47:51.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:47:51.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:47:51.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:47:51.538 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836504 2024-04-25T09:47:51.562 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672983 2024-04-25T09:47:51.586 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509447 2024-04-25T09:47:51.601 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:47:54.603 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:47:54.604 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:47:54.768 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:47:54.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:47:54.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:47:54.769 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:47:54.924 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:47:54.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:47:54.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:47:55.078 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:47:55.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:47:55.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:47:55.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:47:55.230 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:47:55.231 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:47:55.352 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:47:55.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:55.353 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:55.465 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:55.468 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:55.469 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:55.633 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:55.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:55.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:55.750 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:55.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:55.754 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:55.867 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:55.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:55.871 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:56.001 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:56.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:56.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:56.169 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:56.172 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:56.173 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:56.294 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:56.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:56.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:56.412 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:56.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:56.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:56.529 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:56.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:56.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:56.654 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:56.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:56.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:56.813 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:56.816 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:56.817 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:56.931 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:56.935 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:57.048 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:57.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:57.052 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:57.165 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:57.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:57.168 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:57.282 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:57.285 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:57.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:57.399 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:57.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:57.402 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:57.559 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:57.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:57.562 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:57.701 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:57.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:57.704 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:57.826 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:57.829 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:47:57.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:47:58.002 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:47:58.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:47:58.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:47:58.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:47:58.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:47:58.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:47:58.152 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836507 2024-04-25T09:47:58.176 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672987 2024-04-25T09:47:58.199 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509450 2024-04-25T09:47:58.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:01.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:01.217 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:01.380 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T09:48:01.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:01.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:01.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:48:01.540 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T09:48:01.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:01.540 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:01.696 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:01.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:01.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:01.696 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:01.848 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:01.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:124: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-04-25T09:48:01.984 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:48:02.012 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:48:02.045 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:48:02.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:125: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-04-25T09:48:02.224 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:48:02.254 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:48:02.286 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:48:02.301 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:126: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-04-25T09:48:02.438 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:48:02.466 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:48:02.494 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:48:02.509 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:127: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-04-25T09:48:02.646 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:48:02.676 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:48:02.703 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:48:02.717 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:128: TEST_trim_max_entries: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 0 2024-04-25T09:48:02.851 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:48:02.878 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:48:02.909 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:48:02.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:131: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:03.084 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:03.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:132: TEST_trim_max_entries: test_log_size 1.0 18 2024-04-25T09:48:03.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:03.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-04-25T09:48:03.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:03.089 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:03.230 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836510 2024-04-25T09:48:03.253 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672990 2024-04-25T09:48:03.277 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509453 2024-04-25T09:48:03.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:06.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:06.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:06.460 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:48:06.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:06.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:06.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-04-25T09:48:06.614 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:48:06.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:06.615 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:06.755 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:06.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:06.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:06.755 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:06.907 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:06.907 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:133: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:07.053 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:07.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:134: TEST_trim_max_entries: test_log_size 1.0 15 2024-04-25T09:48:07.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:07.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-04-25T09:48:07.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:07.056 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:07.197 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836513 2024-04-25T09:48:07.221 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672993 2024-04-25T09:48:07.244 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509456 2024-04-25T09:48:07.259 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:10.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:10.262 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:10.424 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:48:10.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:10.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:10.425 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-04-25T09:48:10.583 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:48:10.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:10.584 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:10.728 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:10.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:10.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:10.729 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:10.880 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:10.880 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:135: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:10.989 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:10.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:136: TEST_trim_max_entries: test_log_size 1.0 12 2024-04-25T09:48:10.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:10.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-04-25T09:48:10.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:10.992 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:11.135 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836516 2024-04-25T09:48:11.157 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672995 2024-04-25T09:48:11.181 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509459 2024-04-25T09:48:11.195 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:14.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:14.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:14.361 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:48:14.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:14.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:14.362 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-04-25T09:48:14.517 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:48:14.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:14.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:14.675 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:14.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:14.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:14.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:14.836 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:14.836 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:137: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:14.983 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:14.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:138: TEST_trim_max_entries: test_log_size 1.0 9 2024-04-25T09:48:14.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:14.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-04-25T09:48:14.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:14.987 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:15.134 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836519 2024-04-25T09:48:15.158 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672998 2024-04-25T09:48:15.183 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509461 2024-04-25T09:48:15.198 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:18.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:18.201 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:18.364 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:48:18.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:18.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:18.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-04-25T09:48:18.516 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:48:18.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:18.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:18.669 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:18.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:18.670 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:18.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:18.825 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:18.826 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:139: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:18.978 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:18.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:140: TEST_trim_max_entries: test_log_size 1.0 6 2024-04-25T09:48:18.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:18.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-04-25T09:48:18.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:18.982 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:19.126 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836522 2024-04-25T09:48:19.149 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673001 2024-04-25T09:48:19.173 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509464 2024-04-25T09:48:19.188 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:22.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:22.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:22.356 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:48:22.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:22.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-04-25T09:48:22.357 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:22.517 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:48:22.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:22.517 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:22.676 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:22.676 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:22.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:22.677 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:22.830 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:22.830 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:141: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:22.947 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:22.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:142: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:48:22.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:22.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:48:22.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:22.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:23.090 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836524 2024-04-25T09:48:23.113 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673004 2024-04-25T09:48:23.137 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509467 2024-04-25T09:48:23.152 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:26.155 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:26.156 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:26.320 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:48:26.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:26.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:26.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:48:26.473 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:48:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:26.474 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:26.635 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:26.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:26.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:26.635 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:26.789 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:26.789 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:145: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:26.924 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:26.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:146: TEST_trim_max_entries: test_log_size 1.0 4 2024-04-25T09:48:26.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:26.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:48:26.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:26.929 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:27.073 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836527 2024-04-25T09:48:27.096 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673007 2024-04-25T09:48:27.120 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509470 2024-04-25T09:48:27.134 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:30.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:30.137 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:30.301 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:48:30.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:30.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:30.302 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:48:30.461 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:48:30.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:30.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:30.617 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:30.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:30.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:30.618 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:30.770 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:30.771 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:147: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:30.927 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:148: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:48:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:48:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:30.930 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:31.065 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836530 2024-04-25T09:48:31.089 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673009 2024-04-25T09:48:31.113 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509473 2024-04-25T09:48:31.127 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:34.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:34.130 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:34.293 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:48:34.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:34.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:34.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:48:34.452 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:48:34.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:34.453 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:34.609 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:34.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:34.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:34.610 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:34.765 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:34.765 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:149: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:34.871 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:34.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:150: TEST_trim_max_entries: test_log_size 1.0 4 2024-04-25T09:48:34.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:34.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:48:34.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:34.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:35.018 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836533 2024-04-25T09:48:35.042 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673012 2024-04-25T09:48:35.066 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509475 2024-04-25T09:48:35.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:38.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:38.084 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:38.249 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:48:38.249 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:38.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:38.250 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:48:38.404 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:48:38.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:38.405 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:38.551 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:38.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:38.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:38.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:38.707 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:38.707 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:151: TEST_trim_max_entries: rados -p test rm foo 2024-04-25T09:48:38.815 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:48:38.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:152: TEST_trim_max_entries: test_log_size 1.0 3 2024-04-25T09:48:38.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:48:38.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:48:38.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:48:38.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:48:38.961 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836535 2024-04-25T09:48:38.985 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673015 2024-04-25T09:48:39.009 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509478 2024-04-25T09:48:39.023 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:48:42.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:42.025 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:42.189 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:48:42.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:42.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:48:42.189 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:48:42.341 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:48:42.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:42.342 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:42.500 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:42.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:48:42.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:48:42.501 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:48:42.656 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:42.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:48:42.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:48:42.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:48:42.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:48:42.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:48:42.658 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:48:42.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:48:42.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:48:42.658 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:48:42.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:48:42.967 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:48:42.968 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:48:42.969 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:48:42.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:48:42.971 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:48:42.971 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:48:42.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:48:42.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:48:42.972 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:48:42.973 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:48:42.974 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:48:42.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:48:42.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:48:43.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:48:43.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:48:43.000 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:48:43.000 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494734 2024-04-25T09:48:43.002 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:32: run: for func in $funcs 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:33: run: setup td/repro-long-log 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/repro-long-log 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/repro-long-log 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:48:43.003 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:48:43.005 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:48:43.005 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:48:43.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:48:43.006 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:48:43.007 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:48:43.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:48:43.008 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:48:43.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:48:43.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:48:43.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:48:43.009 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:48:43.011 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:48:43.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:48:43.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:48:43.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:48:43.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:48:43.013 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:48:43.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494734 2024-04-25T09:48:43.014 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:48:43.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:48:43.015 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/repro-long-log 2024-04-25T09:48:43.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-04-25T09:48:43.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:48:43.016 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:48:43.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.494734 2024-04-25T09:48:43.019 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-04-25T09:48:43.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-04-25T09:48:43.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/repro-long-log 1' TERM HUP INT 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:34: run: TEST_trim_max_entries_with_dups td/repro-long-log 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:157: TEST_trim_max_entries_with_dups: local dir=td/repro-long-log 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:159: TEST_trim_max_entries_with_dups: setup_log_test td/repro-long-log 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:55: setup_log_test: local dir=td/repro-long-log 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:56: setup_log_test: local which= 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:58: setup_log_test: run_mon td/repro-long-log a 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/repro-long-log 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/repro-long-log/a 2024-04-25T09:48:43.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/repro-long-log/a --run-dir=td/repro-long-log 2024-04-25T09:48:43.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-04-25T09:48:43.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:48:43.520 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:48:43.521 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:48:43.521 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:48:43.521 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:48:43.521 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:48:43.521 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/repro-long-log/a '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --mon-cluster-log-file=td/repro-long-log/log --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-04-25T09:48:43.703 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-04-25T09:48:43.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-04-25T09:48:43.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:48:43.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:48:43.704 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-04-25T09:48:43.705 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-04-25T09:48:43.705 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:48:43.706 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:48:43.706 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:48:43.706 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:48:43.706 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:48:43.706 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:48:43.706 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494734/ceph-mon.a.asok 2024-04-25T09:48:43.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:48:43.707 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494734/ceph-mon.a.asok config get fsid 2024-04-25T09:48:43.821 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-04-25T09:48:43.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-04-25T09:48:43.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-04-25T09:48:43.822 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-04-25T09:48:43.823 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-04-25T09:48:43.823 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-04-25T09:48:43.823 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-04-25T09:48:43.823 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-04-25T09:48:43.823 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-04-25T09:48:43.823 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:48:43.823 INFO:tasks.workunit.client.0.smithi134.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:48:43.824 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.494734/ceph-mon.a.asok 2024-04-25T09:48:43.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-04-25T09:48:43.824 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.494734/ceph-mon.a.asok config get mon_host 2024-04-25T09:48:43.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:59: setup_log_test: run_mgr td/repro-long-log x 2024-04-25T09:48:43.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/repro-long-log 2024-04-25T09:48:43.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-04-25T09:48:43.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-04-25T09:48:43.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-04-25T09:48:43.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/repro-long-log/x 2024-04-25T09:48:43.934 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-04-25T09:48:44.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-04-25T09:48:44.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:48:44.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:48:44.846 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:48:44.846 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:48:44.846 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:48:44.846 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:48:44.847 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:48:44.848 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/repro-long-log/x '--log-file=td/repro-long-log/$name.log' '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --run-dir=td/repro-long-log '--pid-file=td/repro-long-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:60: setup_log_test: run_osd td/repro-long-log 0 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/0 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:48:44.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/0' 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/0/journal' 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:48:44.884 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:48:44.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:48:44.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:48:44.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:48:44.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:48:44.885 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/0 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:48:44.886 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5f51f2c6-c51f-41a3-ba98-a1014cb7e18f 2024-04-25T09:48:44.887 INFO:tasks.workunit.client.0.smithi134.stdout:add osd0 5f51f2c6-c51f-41a3-ba98-a1014cb7e18f 2024-04-25T09:48:44.887 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 5f51f2c6-c51f-41a3-ba98-a1014cb7e18f' 2024-04-25T09:48:44.887 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:48:44.898 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD8JipmUhq8NRAAkCH2/Z5Ea6H8z7gVkDTsTw== 2024-04-25T09:48:44.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD8JipmUhq8NRAAkCH2/Z5Ea6H8z7gVkDTsTw=="}' 2024-04-25T09:48:44.899 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5f51f2c6-c51f-41a3-ba98-a1014cb7e18f -i td/repro-long-log/0/new.json 2024-04-25T09:48:45.218 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:45.234 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/0/new.json 2024-04-25T09:48:45.236 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8JipmUhq8NRAAkCH2/Z5Ea6H8z7gVkDTsTw== --osd-uuid 5f51f2c6-c51f-41a3-ba98-a1014cb7e18f 2024-04-25T09:48:45.263 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:45.263+0000 7f8d9a4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:48:45.286 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:45.287+0000 7f8d9a4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:48:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:45.287+0000 7f8d9a4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:48:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:45.287+0000 7f8d9a4e27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:48:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:45.287+0000 7f8d9a4e27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:48:45.288 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:45.287+0000 7f8d9a4e27c0 -1 bluestore(td/repro-long-log/0/block) _read_bdev_label failed to open td/repro-long-log/0/block: (2) No such file or directory 2024-04-25T09:48:45.326 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:45.327+0000 7f8d9a4e27c0 -1 bluestore(td/repro-long-log/0) _read_fsid unparsable uuid 2024-04-25T09:48:51.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/0/keyring 2024-04-25T09:48:51.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:48:51.742 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd0 key to auth repository 2024-04-25T09:48:51.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-04-25T09:48:51.742 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:48:52.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-04-25T09:48:52.320 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.0 2024-04-25T09:48:52.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/0 --osd-journal=td/repro-long-log/0/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:48:52.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:48:52.321 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:48:52.322 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:48:52.349 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:52.347+0000 7f1730f207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:48:52.359 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:52.359+0000 7f1730f207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:48:52.360 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:52.359+0000 7f1730f207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:48:52.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-04-25T09:48:52.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:48:52.732 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-04-25T09:48:52.733 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:48:52.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:48:52.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:48:52.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:48:52.733 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:48:52.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:48:52.734 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:48:52.935 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:52.935+0000 7f1730f207c0 -1 Falling back to public interface 2024-04-25T09:48:53.178 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:48:54.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:48:54.179 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:48:54.180 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:48:54.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:48:54.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:48:54.180 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:48:54.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:48:55.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:48:55.591 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:48:55.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:48:55.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:48:55.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:48:55.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:48:55.593 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:48:55.591+0000 7f1730f207c0 -1 osd.0 0 log_to_monitors true 2024-04-25T09:48:56.107 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:48:57.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:48:57.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:48:57.109 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:48:57.109 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:48:57.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:48:57.110 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:48:57.588 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:48:58.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:48:58.590 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:48:58.590 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:48:58.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:48:58.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:48:58.591 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:48:59.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:00.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:00.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:00.027 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:49:00.028 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:49:00.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:00.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:00.542 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:01.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:01.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:01.544 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:49:01.544 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:49:01.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:01.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:02.010 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:03.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:03.012 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:03.012 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:49:03.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:49:03.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:03.013 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:03.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:04.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:04.548 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:04.549 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:49:04.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:49:04.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:04.549 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:04.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:05.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:05.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:05.980 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:49:05.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:49:05.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:05.981 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:06.456 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:07.457 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:07.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:07.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:49:07.458 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:49:07.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:07.458 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:07.872 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:08.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:08.874 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:08.874 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:49:08.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:49:08.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:08.875 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:09.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:10.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:10.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:10.484 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:49:10.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:49:10.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:10.485 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:11.264 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:12.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:12.266 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:12.266 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:49:12.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:49:12.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:12.267 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:13.413 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:14.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:14.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:14.415 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:49:14.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:49:14.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:14.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:15.574 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:16.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:16.576 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:16.576 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:49:16.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:49:16.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:16.577 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:17.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:18.440 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:18.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:18.441 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:49:18.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:49:18.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:18.441 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:18.995 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:19.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:19.997 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:19.997 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:49:19.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:49:19.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:19.998 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-04-25T09:49:20.477 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/8297688,v1:127.0.0.1:6803/8297688] [v2:127.0.0.1:6804/8297688,v1:127.0.0.1:6805/8297688] exists,up 5f51f2c6-c51f-41a3-ba98-a1014cb7e18f 2024-04-25T09:49:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:49:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:49:20.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:61: setup_log_test: run_osd td/repro-long-log 1 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/1 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/1' 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/1/journal' 2024-04-25T09:49:20.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:49:20.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:49:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:49:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:49:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:49:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:49:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:49:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:49:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:49:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:49:20.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/1 2024-04-25T09:49:20.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:49:20.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=898dfe3b-09b1-42f8-b4a5-1a8b6158d630 2024-04-25T09:49:20.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 898dfe3b-09b1-42f8-b4a5-1a8b6158d630' 2024-04-25T09:49:20.484 INFO:tasks.workunit.client.0.smithi134.stdout:add osd1 898dfe3b-09b1-42f8-b4a5-1a8b6158d630 2024-04-25T09:49:20.484 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:49:20.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAgJypmOJXlHRAADkzR1gdIHCIBSg2TnSdLMw== 2024-04-25T09:49:20.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAgJypmOJXlHRAADkzR1gdIHCIBSg2TnSdLMw=="}' 2024-04-25T09:49:20.499 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 898dfe3b-09b1-42f8-b4a5-1a8b6158d630 -i td/repro-long-log/1/new.json 2024-04-25T09:49:20.993 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:49:21.008 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/1/new.json 2024-04-25T09:49:21.009 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAgJypmOJXlHRAADkzR1gdIHCIBSg2TnSdLMw== --osd-uuid 898dfe3b-09b1-42f8-b4a5-1a8b6158d630 2024-04-25T09:49:21.040 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:21.039+0000 7fad43b7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:49:21.067 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:21.067+0000 7fad43b7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:49:21.068 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:21.067+0000 7fad43b7e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:49:21.069 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:21.067+0000 7fad43b7e7c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:49:21.069 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:21.067+0000 7fad43b7e7c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:49:21.069 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:21.067+0000 7fad43b7e7c0 -1 bluestore(td/repro-long-log/1/block) _read_bdev_label failed to open td/repro-long-log/1/block: (2) No such file or directory 2024-04-25T09:49:21.106 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:21.107+0000 7fad43b7e7c0 -1 bluestore(td/repro-long-log/1) _read_fsid unparsable uuid 2024-04-25T09:49:27.852 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/1/keyring 2024-04-25T09:49:27.853 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:49:27.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-04-25T09:49:27.854 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd1 key to auth repository 2024-04-25T09:49:27.854 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:49:28.415 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-04-25T09:49:28.415 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.1 2024-04-25T09:49:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/1 --osd-journal=td/repro-long-log/1/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:49:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:49:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:49:28.416 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:49:28.444 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:28.442+0000 7fbc269ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:49:28.447 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:28.446+0000 7fbc269ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:49:28.448 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:28.446+0000 7fbc269ba7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:49:28.818 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-04-25T09:49:28.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:49:28.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-04-25T09:49:28.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:49:28.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:49:28.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:28.819 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:49:28.819 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:49:28.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:28.820 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:29.000 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:28.998+0000 7fbc269ba7c0 -1 Falling back to public interface 2024-04-25T09:49:29.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:30.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:30.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:30.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:49:30.289 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:49:30.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:30.290 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:30.689 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:31.626 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:31.622+0000 7fbc269ba7c0 -1 osd.1 0 log_to_monitors true 2024-04-25T09:49:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:31.691 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:49:31.691 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:49:31.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:31.692 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:32.334 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:33.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:33.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:33.336 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:49:33.336 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:49:33.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:33.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:33.946 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:34.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:34.948 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:34.948 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:49:34.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:49:34.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:34.949 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:35.383 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:36.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:36.385 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:36.385 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:49:36.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:49:36.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:36.386 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:36.859 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:37.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:37.860 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:37.861 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:49:37.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:49:37.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:37.861 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:38.432 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:39.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:39.435 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:39.435 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:49:39.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:49:39.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:39.436 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:39.938 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:40.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:40.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:40.941 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:49:40.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:49:40.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:40.941 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:41.374 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:42.375 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:42.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:42.376 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:49:42.376 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:49:42.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:42.377 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:42.844 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:43.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:43.846 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:43.846 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:49:43.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:49:43.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:43.847 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:44.323 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:45.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:45.325 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:45.326 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:49:45.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:49:45.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:45.326 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:46.016 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:47.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:47.018 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:47.018 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:49:47.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:49:47.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:47.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:47.794 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:48.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:48.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:48.796 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:49:48.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:49:48.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:48.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:49.550 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:50.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:50.552 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:50.552 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:49:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:49:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:50.553 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:51.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:52.368 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:52.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:52.369 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:49:52.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:49:52.369 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:52.370 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:53.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:54.272 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:54.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:54.273 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:49:54.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:49:54.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:54.273 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:55.481 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:56.482 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:56.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:56.483 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:49:56.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:49:56.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:56.484 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:57.019 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:49:58.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:49:58.020 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:49:58.021 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:49:58.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-04-25T09:49:58.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:49:58.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-04-25T09:49:58.440 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2876422764,v1:127.0.0.1:6811/2876422764] [v2:127.0.0.1:6812/2876422764,v1:127.0.0.1:6813/2876422764] exists,up 898dfe3b-09b1-42f8-b4a5-1a8b6158d630 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:62: setup_log_test: run_osd td/repro-long-log 2 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/repro-long-log 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/repro-long-log/2 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 ' 2024-04-25T09:49:58.445 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/repro-long-log/2' 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/repro-long-log/2/journal' 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/repro-long-log' 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-04-25T09:49:58.446 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/repro-long-log/$name.log' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/repro-long-log/$name.pid' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-04-25T09:49:58.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-04-25T09:49:58.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-04-25T09:49:58.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-04-25T09:49:58.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-04-25T09:49:58.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/repro-long-log/2 2024-04-25T09:49:58.448 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-04-25T09:49:58.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b64caa4b-78d2-4c08-b306-caf2fd17f46d 2024-04-25T09:49:58.448 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 b64caa4b-78d2-4c08-b306-caf2fd17f46d' 2024-04-25T09:49:58.449 INFO:tasks.workunit.client.0.smithi134.stdout:add osd2 b64caa4b-78d2-4c08-b306-caf2fd17f46d 2024-04-25T09:49:58.449 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-04-25T09:49:58.460 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBGJypmkPCUGxAASpunJFnOmHq8lFx4dYJPUg== 2024-04-25T09:49:58.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBGJypmkPCUGxAASpunJFnOmHq8lFx4dYJPUg=="}' 2024-04-25T09:49:58.461 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b64caa4b-78d2-4c08-b306-caf2fd17f46d -i td/repro-long-log/2/new.json 2024-04-25T09:49:58.960 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:49:58.976 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/repro-long-log/2/new.json 2024-04-25T09:49:58.978 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBGJypmkPCUGxAASpunJFnOmHq8lFx4dYJPUg== --osd-uuid b64caa4b-78d2-4c08-b306-caf2fd17f46d 2024-04-25T09:49:59.004 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:59.002+0000 7f69632cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:49:59.037 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:59.034+0000 7f69632cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:49:59.038 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:59.034+0000 7f69632cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:49:59.038 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:59.034+0000 7f69632cb7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:49:59.038 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:59.034+0000 7f69632cb7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:49:59.038 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:59.034+0000 7f69632cb7c0 -1 bluestore(td/repro-long-log/2/block) _read_bdev_label failed to open td/repro-long-log/2/block: (2) No such file or directory 2024-04-25T09:49:59.082 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:49:59.078+0000 7f69632cb7c0 -1 bluestore(td/repro-long-log/2) _read_fsid unparsable uuid 2024-04-25T09:50:05.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/repro-long-log/2/keyring 2024-04-25T09:50:05.812 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-04-25T09:50:05.814 INFO:tasks.workunit.client.0.smithi134.stdout:adding osd2 key to auth repository 2024-04-25T09:50:05.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-04-25T09:50:05.814 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/repro-long-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-04-25T09:50:06.418 INFO:tasks.workunit.client.0.smithi134.stdout:start osd.2 2024-04-25T09:50:06.418 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-04-25T09:50:06.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=59e7131b-6e76-489d-88dd-6c56eabe5385 --auth-supported=none --mon-host=127.0.0.1:7100 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/repro-long-log/2 --osd-journal=td/repro-long-log/2/journal --chdir= --run-dir=td/repro-long-log '--admin-socket=/tmp/ceph-asok.494734/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/repro-long-log/$name.log' '--pid-file=td/repro-long-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-04-25T09:50:06.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-04-25T09:50:06.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-04-25T09:50:06.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-04-25T09:50:06.442 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:50:06.442+0000 7f1b3b57b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:50:06.444 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:50:06.446+0000 7f1b3b57b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:50:06.445 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:50:06.446+0000 7f1b3b57b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-04-25T09:50:06.862 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-04-25T09:50:06.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-04-25T09:50:06.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-04-25T09:50:06.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-04-25T09:50:06.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-04-25T09:50:06.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:06.863 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-04-25T09:50:06.863 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:50:06.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:06.864 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:07.020 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:50:07.022+0000 7f1b3b57b7c0 -1 Falling back to public interface 2024-04-25T09:50:07.286 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:08.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:08.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:08.288 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-04-25T09:50:08.288 INFO:tasks.workunit.client.0.smithi134.stdout:1 2024-04-25T09:50:08.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:08.289 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:08.697 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:09.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:09.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:09.699 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-04-25T09:50:09.700 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:50:09.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:09.700 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:09.708 INFO:tasks.workunit.client.0.smithi134.stderr:2024-04-25T09:50:09.709+0000 7f1b3b57b7c0 -1 osd.2 0 log_to_monitors true 2024-04-25T09:50:10.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:11.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:11.296 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:11.296 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:50:11.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-04-25T09:50:11.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:11.297 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:11.923 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:12.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:12.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:12.925 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-04-25T09:50:12.925 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:50:12.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:12.926 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:13.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:14.392 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:14.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:14.393 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-04-25T09:50:14.393 INFO:tasks.workunit.client.0.smithi134.stdout:5 2024-04-25T09:50:14.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:14.394 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:14.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:15.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:15.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:15.841 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-04-25T09:50:15.841 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:50:15.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:15.842 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:16.389 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:17.390 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:17.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:17.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-04-25T09:50:17.391 INFO:tasks.workunit.client.0.smithi134.stdout:7 2024-04-25T09:50:17.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:17.391 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:18.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:19.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:19.031 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:19.031 INFO:tasks.workunit.client.0.smithi134.stdout:8 2024-04-25T09:50:19.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-04-25T09:50:19.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:19.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:19.743 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:20.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:20.745 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:20.745 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:50:20.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-04-25T09:50:20.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:20.746 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:21.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:22.190 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:22.191 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:50:22.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:22.191 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-04-25T09:50:22.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:22.192 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:22.651 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:23.653 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:23.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:23.654 INFO:tasks.workunit.client.0.smithi134.stdout:11 2024-04-25T09:50:23.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-04-25T09:50:23.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:23.654 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:24.902 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:25.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:25.904 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:25.904 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:50:25.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-04-25T09:50:25.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:25.905 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:27.159 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:28.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:28.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:28.161 INFO:tasks.workunit.client.0.smithi134.stdout:13 2024-04-25T09:50:28.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-04-25T09:50:28.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:28.162 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:28.918 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:29.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:29.920 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:29.920 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:50:29.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-04-25T09:50:29.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:29.921 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:31.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:32.063 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:32.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:32.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-04-25T09:50:32.064 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:50:32.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:32.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:32.477 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:33.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:33.479 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:33.479 INFO:tasks.workunit.client.0.smithi134.stdout:16 2024-04-25T09:50:33.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-04-25T09:50:33.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:33.480 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:34.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-04-25T09:50:35.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-04-25T09:50:35.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-04-25T09:50:35.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-04-25T09:50:35.612 INFO:tasks.workunit.client.0.smithi134.stdout:17 2024-04-25T09:50:35.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-04-25T09:50:35.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-04-25T09:50:36.079 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1074840623,v1:127.0.0.1:6819/1074840623] [v2:127.0.0.1:6820/1074840623,v1:127.0.0.1:6821/1074840623] exists,up b64caa4b-78d2-4c08-b306-caf2fd17f46d 2024-04-25T09:50:36.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-04-25T09:50:36.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-04-25T09:50:36.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-04-25T09:50:36.080 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:64: setup_log_test: ceph osd pool create test 1 1 2024-04-25T09:50:36.635 INFO:tasks.workunit.client.0.smithi134.stderr:pool 'test' already exists 2024-04-25T09:50:36.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: ceph osd dump --format json 2024-04-25T09:50:36.653 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: jq '.pools[] | select(.pool_name == "test") | .pool' 2024-04-25T09:50:37.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:65: setup_log_test: POOL_ID=1 2024-04-25T09:50:37.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:66: setup_log_test: PGID=1.0 2024-04-25T09:50:37.072 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:69: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 20 2024-04-25T09:50:37.269 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:37.312 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:37.348 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:37.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:70: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 20 2024-04-25T09:50:37.515 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:37.544 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:37.572 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:37.587 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:71: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-max-pg-log-entries 30 2024-04-25T09:50:37.726 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:37.754 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:37.782 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:37.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:72: setup_log_test: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 10 2024-04-25T09:50:38.056 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:38.099 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:38.141 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:38.161 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:73: setup_log_test: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:50:38.304 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:38.334 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:38.363 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:38.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:75: setup_log_test: touch td/repro-long-log/foo 2024-04-25T09:50:38.379 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: seq 1 20 2024-04-25T09:50:38.380 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:38.381 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:38.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:38.636 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:38.796 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:38.797 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:39.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:39.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:39.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:39.220 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:39.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:39.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:39.462 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:39.463 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:39.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:39.665 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:39.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:39.822 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:39.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:39.940 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:40.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:40.116 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:40.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:40.241 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:40.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:40.384 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:40.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:40.500 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:40.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:40.617 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:40.776 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:40.777 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:40.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:40.919 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:41.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:41.036 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:41.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:41.153 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:41.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:76: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:41.270 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:78: setup_log_test: rados -p test put foo td/repro-long-log/foo 2024-04-25T09:50:41.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:81: setup_log_test: test_log_size 1.0 20 2024-04-25T09:50:41.387 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:50:41.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=20 2024-04-25T09:50:41.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:50:41.388 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:50:41.545 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836504 2024-04-25T09:50:41.569 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672977 2024-04-25T09:50:41.594 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509452 2024-04-25T09:50:41.609 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:50:44.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:44.612 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:50:44.779 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:50:44.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:44.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:50:44.780 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 20 2024-04-25T09:50:44.933 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:50:44.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:44.933 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:50:45.081 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:50:45.081 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:45.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:50:45.082 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:50:45.231 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:50:45.232 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:83: setup_log_test: rados -p test rm foo 2024-04-25T09:50:45.364 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: seq 1 20 2024-04-25T09:50:45.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:45.365 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:45.528 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:45.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:45.532 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:45.654 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:45.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:45.657 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:45.779 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:45.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:45.782 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:45.888 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:45.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:45.892 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:46.014 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:46.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:46.017 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:46.190 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:46.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:46.194 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:46.307 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:46.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:46.312 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:46.424 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:46.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:46.428 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:46.541 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:46.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:46.545 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:46.718 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:46.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:46.721 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:46.834 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:46.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:46.838 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:46.976 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:46.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:46.980 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:47.093 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:47.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:47.097 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:47.211 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:47.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:47.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:47.344 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:47.347 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:47.348 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:47.528 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:47.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:47.646 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:47.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:47.649 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:47.763 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:47.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:47.766 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:47.880 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:47.883 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:86: setup_log_test: for i in $(seq 1 20) 2024-04-25T09:50:47.884 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:88: setup_log_test: rados -p test rm foo 2024-04-25T09:50:48.047 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:48.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:92: setup_log_test: test_log_size 1.0 21 2024-04-25T09:50:48.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:50:48.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=21 2024-04-25T09:50:48.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=0 2024-04-25T09:50:48.051 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:50:48.194 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836507 2024-04-25T09:50:48.218 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672980 2024-04-25T09:50:48.243 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509455 2024-04-25T09:50:48.258 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:50:51.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:51.261 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:50:51.425 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T09:50:51.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:51.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:50:51.426 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 21 2024-04-25T09:50:51.582 INFO:tasks.workunit.client.0.smithi134.stdout:21 2024-04-25T09:50:51.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:51.583 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:50:51.738 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:50:51.738 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:51.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:50:51.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 0 2024-04-25T09:50:51.897 INFO:tasks.workunit.client.0.smithi134.stdout:0 2024-04-25T09:50:51.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:161: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_target_pg_log_entries_per_osd 2 2024-04-25T09:50:52.041 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:52.068 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:52.096 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:52.111 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:162: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-min-pg-log-entries 2 2024-04-25T09:50:52.259 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:52.289 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:52.321 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:52.337 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:163: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-min 2 2024-04-25T09:50:52.472 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:52.503 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:52.532 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:52.547 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:164: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd-pg-log-trim-max 4 2024-04-25T09:50:52.684 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:52.716 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:52.745 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:52.759 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:165: TEST_trim_max_entries_with_dups: ceph tell 'osd.*' injectargs -- --osd_pg_log_dups_tracked 20 2024-04-25T09:50:52.892 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: {} 2024-04-25T09:50:52.920 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: {} 2024-04-25T09:50:52.952 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: {} 2024-04-25T09:50:52.967 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:169: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:50:53.112 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:53.118 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:170: TEST_trim_max_entries_with_dups: test_log_size 1.0 18 2 2024-04-25T09:50:53.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:50:53.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=18 2024-04-25T09:50:53.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=2 2024-04-25T09:50:53.119 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:50:53.254 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836510 2024-04-25T09:50:53.277 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672983 2024-04-25T09:50:53.302 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509458 2024-04-25T09:50:53.316 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:50:56.319 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:56.320 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:50:56.482 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:50:56.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:50:56.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 18 2024-04-25T09:50:56.483 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:56.637 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:50:56.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:56.637 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:50:56.791 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:50:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:50:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:50:56.791 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 2 2024-04-25T09:50:56.950 INFO:tasks.workunit.client.0.smithi134.stdout:2 2024-04-25T09:50:56.950 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:171: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:50:57.082 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:50:57.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:172: TEST_trim_max_entries_with_dups: test_log_size 1.0 15 6 2024-04-25T09:50:57.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:50:57.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=15 2024-04-25T09:50:57.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=6 2024-04-25T09:50:57.085 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:50:57.230 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836513 2024-04-25T09:50:57.254 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672986 2024-04-25T09:50:57.277 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509461 2024-04-25T09:50:57.292 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:51:00.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:00.295 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:00.459 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:51:00.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:00.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:00.459 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 15 2024-04-25T09:51:00.611 INFO:tasks.workunit.client.0.smithi134.stdout:15 2024-04-25T09:51:00.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:00.611 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:00.761 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:51:00.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:00.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:00.761 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 6 2024-04-25T09:51:00.910 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:51:00.910 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:173: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:51:01.068 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:51:01.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:174: TEST_trim_max_entries_with_dups: test_log_size 1.0 12 10 2024-04-25T09:51:01.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:51:01.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=12 2024-04-25T09:51:01.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=10 2024-04-25T09:51:01.071 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:51:01.209 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836516 2024-04-25T09:51:01.233 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672989 2024-04-25T09:51:01.257 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509463 2024-04-25T09:51:01.271 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:51:04.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:04.274 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:04.439 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:51:04.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:04.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:04.439 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 12 2024-04-25T09:51:04.597 INFO:tasks.workunit.client.0.smithi134.stdout:12 2024-04-25T09:51:04.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:04.597 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:04.747 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:51:04.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:04.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:04.748 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 10 2024-04-25T09:51:04.891 INFO:tasks.workunit.client.0.smithi134.stdout:10 2024-04-25T09:51:04.891 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:175: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:51:05.045 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:51:05.048 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:176: TEST_trim_max_entries_with_dups: test_log_size 1.0 9 14 2024-04-25T09:51:05.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:51:05.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=9 2024-04-25T09:51:05.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=14 2024-04-25T09:51:05.049 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:51:05.185 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836518 2024-04-25T09:51:05.208 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672992 2024-04-25T09:51:05.232 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509466 2024-04-25T09:51:05.245 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:51:08.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:08.248 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:08.410 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:51:08.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:08.410 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:08.411 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 9 2024-04-25T09:51:08.566 INFO:tasks.workunit.client.0.smithi134.stdout:9 2024-04-25T09:51:08.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:08.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:08.726 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:51:08.726 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:08.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:08.727 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 14 2024-04-25T09:51:08.877 INFO:tasks.workunit.client.0.smithi134.stdout:14 2024-04-25T09:51:08.878 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:177: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:51:09.040 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:51:09.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:178: TEST_trim_max_entries_with_dups: test_log_size 1.0 6 18 2024-04-25T09:51:09.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:51:09.043 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=6 2024-04-25T09:51:09.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=18 2024-04-25T09:51:09.044 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:51:09.189 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836521 2024-04-25T09:51:09.213 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672995 2024-04-25T09:51:09.238 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509469 2024-04-25T09:51:09.252 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:51:12.256 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:12.257 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:12.418 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:51:12.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:12.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 6 2024-04-25T09:51:12.419 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:12.580 INFO:tasks.workunit.client.0.smithi134.stdout:6 2024-04-25T09:51:12.580 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:12.581 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:12.739 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:51:12.739 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:12.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:12.740 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 18 2024-04-25T09:51:12.897 INFO:tasks.workunit.client.0.smithi134.stdout:18 2024-04-25T09:51:12.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:179: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:51:13.017 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:51:13.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:180: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:51:13.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:51:13.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:51:13.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:51:13.021 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:51:13.163 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836524 2024-04-25T09:51:13.187 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949672997 2024-04-25T09:51:13.211 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509472 2024-04-25T09:51:13.226 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:51:16.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:16.229 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:16.394 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:51:16.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:16.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:16.395 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:51:16.551 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:51:16.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:16.551 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:16.705 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:16.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:16.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:16.705 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:51:16.855 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:16.856 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:183: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:51:17.002 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:51:17.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:184: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-04-25T09:51:17.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:51:17.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:51:17.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:51:17.006 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:51:17.148 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836527 2024-04-25T09:51:17.172 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673000 2024-04-25T09:51:17.197 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509475 2024-04-25T09:51:17.212 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:51:20.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:20.214 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:20.378 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:51:20.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:20.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:20.378 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:51:20.533 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:51:20.533 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:20.534 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:20.686 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:20.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:20.686 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:20.687 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:51:20.843 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:20.843 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:185: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:51:20.989 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:51:20.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:186: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:51:20.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:51:20.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:51:20.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:51:20.993 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:51:21.135 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836530 2024-04-25T09:51:21.159 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673003 2024-04-25T09:51:21.184 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509477 2024-04-25T09:51:21.199 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:51:24.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:24.202 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:24.365 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:51:24.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:24.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:51:24.366 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:24.518 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:51:24.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:24.519 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:24.671 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:24.671 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:24.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:24.672 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:51:24.825 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:24.825 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:187: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:51:24.924 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:51:24.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:188: TEST_trim_max_entries_with_dups: test_log_size 1.0 4 20 2024-04-25T09:51:24.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:51:24.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=4 2024-04-25T09:51:24.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:51:24.928 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:51:25.065 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836532 2024-04-25T09:51:25.089 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673006 2024-04-25T09:51:25.113 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509480 2024-04-25T09:51:25.126 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:51:28.128 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:28.129 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:28.293 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:51:28.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:28.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:28.294 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 4 2024-04-25T09:51:28.446 INFO:tasks.workunit.client.0.smithi134.stdout:4 2024-04-25T09:51:28.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:28.447 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:28.591 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:28.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:28.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:28.592 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:51:28.737 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:28.737 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:189: TEST_trim_max_entries_with_dups: rados -p test rm foo 2024-04-25T09:51:28.894 INFO:tasks.workunit.client.0.smithi134.stderr:error removing test>foo: (2) No such file or directory 2024-04-25T09:51:28.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:190: TEST_trim_max_entries_with_dups: test_log_size 1.0 3 20 2024-04-25T09:51:28.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:43: test_log_size: local PGID=1.0 2024-04-25T09:51:28.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:44: test_log_size: local EXPECTED=3 2024-04-25T09:51:28.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:45: test_log_size: local DUPS_EXPECTED=20 2024-04-25T09:51:28.897 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:46: test_log_size: ceph tell 'osd.*' flush_pg_stats 2024-04-25T09:51:29.038 INFO:tasks.workunit.client.0.smithi134.stdout:osd.0: 21474836535 2024-04-25T09:51:29.060 INFO:tasks.workunit.client.0.smithi134.stdout:osd.1: 42949673009 2024-04-25T09:51:29.083 INFO:tasks.workunit.client.0.smithi134.stdout:osd.2: 64424509483 2024-04-25T09:51:29.096 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:47: test_log_size: sleep 3 2024-04-25T09:51:32.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:32.099 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:48: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:32.262 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:51:32.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:32.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: jq .info.stats.log_size 2024-04-25T09:51:32.263 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:49: test_log_size: grep 3 2024-04-25T09:51:32.421 INFO:tasks.workunit.client.0.smithi134.stdout:3 2024-04-25T09:51:32.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:32.422 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:50: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:32.566 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:32.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: ceph pg 1.0 query 2024-04-25T09:51:32.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: jq .info.stats.log_dups_size 2024-04-25T09:51:32.567 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:51: test_log_size: grep 20 2024-04-25T09:51:32.713 INFO:tasks.workunit.client.0.smithi134.stdout:20 2024-04-25T09:51:32.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/repro_long_log.sh:35: run: teardown td/repro-long-log 2024-04-25T09:51:32.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:51:32.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-04-25T09:51:32.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:51:32.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:51:32.714 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:51:32.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:51:32.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:51:32.714 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:51:33.024 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:51:33.024 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:51:33.026 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:51:33.026 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:51:33.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:51:33.028 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:51:33.028 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:51:33.029 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:51:33.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:51:33.030 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:51:33.030 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:51:33.032 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:51:33.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-04-25T09:51:33.033 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:51:33.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:51:33.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:51:33.055 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:51:33.055 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494734 2024-04-25T09:51:33.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:51:33.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:51:33.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2330: main: code=0 2024-04-25T09:51:33.057 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2334: main: teardown td/repro-long-log 0 2024-04-25T09:51:33.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/repro-long-log 2024-04-25T09:51:33.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-04-25T09:51:33.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/repro-long-log KILL 2024-04-25T09:51:33.058 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-04-25T09:51:33.058 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-04-25T09:51:33.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-04-25T09:51:33.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-04-25T09:51:33.058 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-04-25T09:51:33.060 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-04-25T09:51:33.060 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-04-25T09:51:33.061 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-04-25T09:51:33.061 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-04-25T09:51:33.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-04-25T09:51:33.062 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-04-25T09:51:33.063 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-04-25T09:51:33.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:51:33.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-04-25T09:51:33.064 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-04-25T09:51:33.064 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-25T09:51:33.065 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:51:33.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-04-25T09:51:33.066 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/repro-long-log 2024-04-25T09:51:33.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-04-25T09:51:33.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-04-25T09:51:33.068 INFO:tasks.workunit.client.0.smithi134.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.494734 2024-04-25T09:51:33.068 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.494734 2024-04-25T09:51:33.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-04-25T09:51:33.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-04-25T09:51:33.069 INFO:tasks.workunit.client.0.smithi134.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2335: main: return 0 2024-04-25T09:51:33.070 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-25T09:51:33.071 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-25T09:51:33.084 INFO:tasks.workunit:Stopping ['osd'] on client.0... 2024-04-25T09:51:33.084 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-25T09:51:33.445 DEBUG:teuthology.parallel:result is None 2024-04-25T09:51:33.446 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:51:33.455 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:51:33.455 DEBUG:teuthology.orchestra.run.smithi134:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-04-25T09:51:33.520 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-04-25T09:51:33.521 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-25T09:51:33.532 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-25T09:51:33.532 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-25T09:51:33.604 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-25T09:51:33.612 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-25T09:51:33.612 DEBUG:teuthology.orchestra.run.smithi134:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-25T09:51:33.636 INFO:teuthology.orchestra.run.smithi134.stdout: remote refid st t when poll reach delay offset jitter 2024-04-25T09:51:33.636 INFO:teuthology.orchestra.run.smithi134.stdout:============================================================================== 2024-04-25T09:51:33.636 INFO:teuthology.orchestra.run.smithi134.stdout:+hv01.front.sepi 67.205.162.81 3 u 186 256 377 0.175 -2.689 0.788 2024-04-25T09:51:33.636 INFO:teuthology.orchestra.run.smithi134.stdout:+hv02.front.sepi 63.231.80.2 3 u 192 256 377 0.173 -0.091 0.736 2024-04-25T09:51:33.636 INFO:teuthology.orchestra.run.smithi134.stdout:*hv03.front.sepi 74.6.168.72 3 u 68 256 377 0.183 -2.052 0.562 2024-04-25T09:51:33.636 INFO:teuthology.orchestra.run.smithi134.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-04-25T09:51:33.637 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-25T09:51:33.645 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-25T09:51:33.646 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-25T09:51:33.654 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-25T09:51:33.663 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-25T09:51:33.671 INFO:teuthology.task.internal:Duration was 10535.790200 seconds 2024-04-25T09:51:33.671 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-25T09:51:33.680 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-25T09:51:33.680 DEBUG:teuthology.orchestra.run.smithi134:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-25T09:51:33.720 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-25T09:51:33.720 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi134.front.sepia.ceph.com 2024-04-25T09:51:33.720 DEBUG:teuthology.orchestra.run.smithi134:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-25T09:51:33.773 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-25T09:51:33.773 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-25T09:51:33.847 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-25T09:51:33.847 DEBUG:teuthology.orchestra.run.smithi134:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-25T09:51:33.945 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-25T09:51:33.954 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-25T09:51:33.955 DEBUG:teuthology.orchestra.run.smithi134:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-25T09:51:33.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-25T09:51:33.977 DEBUG:teuthology.orchestra.run.smithi134:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:51:34.022 INFO:teuthology.orchestra.run.smithi134.stdout:kernel.core_pattern = core 2024-04-25T09:51:34.035 DEBUG:teuthology.orchestra.run.smithi134:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-25T09:51:34.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-25T09:51:34.084 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-25T09:51:34.093 INFO:teuthology.task.internal:Transferring archived files... 2024-04-25T09:51:34.095 DEBUG:teuthology.misc:Transferring archived files from smithi134:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/nmordech-2024-04-25_06:28:28-rados:standalone-main-distro-default-smithi/7672922/remote/smithi134 2024-04-25T09:51:34.097 DEBUG:teuthology.orchestra.run.smithi134:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-25T09:51:34.148 INFO:teuthology.task.internal:Removing archive directory... 2024-04-25T09:51:34.148 DEBUG:teuthology.orchestra.run.smithi134:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-25T09:51:34.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-25T09:51:34.204 INFO:teuthology.task.internal:Not uploading archives. 2024-04-25T09:51:34.205 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-25T09:51:34.219 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-25T09:51:34.219 DEBUG:teuthology.orchestra.run.smithi134:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-25T09:51:34.240 INFO:teuthology.orchestra.run.smithi134.stdout: 658641 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 25 09:51 /home/ubuntu/cephtest 2024-04-25T09:51:34.241 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-25T09:51:34.254 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-25T09:51:34.286 INFO:teuthology.run:Summary data: description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} duration: 10535.790199756622 flavor: default owner: scheduled_nmordech@teuthology success: true 2024-04-25T09:51:34.286 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-25T09:51:34.359 INFO:teuthology.run:pass